## Motivation
The `CurrentThread` runtime's `Handle` is different than the
`CurrentThread` executor's `Handle`. This causes interoperability issues
with custom runtimes that build on the `CurrentThread` executor.
Actix-rt offers a concrete example of the issue: [`System::run_in_executor`][1]
requires a `CurrentThread` executor handle - the change in this PR
allows a `CurrentThread` runtime to be used here.
## Solution
This PR adds `fn into_inner(self)` on the runtime `Handle` that consumes
it and returns the underlying `CurrentThread` executor's `Handle`.
[1]: https://docs.rs/actix-rt/0.2.6/actix_rt/struct.System.html#method.run_in_executor
tokio-uds 0.2.4 has the `UnixDatagramFramed` which is reexported here
and tokio-threadpool 0.1.16 uses a new enough rand that compiles with
modern toolchains.
## Motivation
The initial version of `tokio-compat`'s compatibility runtime added in
#1663 doesn't support the calls to `tokio_threadpool` 0.1's `blocking`.
This is because (unlike the timer, executor, and reactor), there's no
way to override the global `blocking` functionality in
`tokio-threadpool`.
## Solution
As discussed [here][1], this branch adds APIs to the v0.1.x version of
`tokio-threadpool` that allow overriding the behavior used by calls to
`blocking`. The threadpool crate now exposes `blocking::set_default` and
`blocking::with_default` functions, like `executor`, `timer`, and
`reactor`. This will allow `tokio-compat` to override calls to 0.1's
`blocking` to use the new `tokio` 0.2 blocking APIs.
Unlike the similar APIs in `executor`, `timer`, and `reactor`, the hooks
for overriding blocking behaviour are `#[doc(hidden)]` and have comments
warning against their use outside of `tokio-compat`. In general, there
probably won't be a compelling reason to override these outside of the
compatibility layer.
Refs: #1722
[1]: https://github.com/tokio-rs/tokio/pull/1663#issuecomment-548661766
Signed-off-by: Eliza Weisman <[email protected]>
This commit adds `set_default` drop guard style APIs for setting the
default reactor, executor, and timer. These are similar to the APIs used
in `tokio` 0.2
In addition to having potentially better ergonomics than the
`with_default` closure APIs, the drop-guard based APIs will be helpful
in rewriting the `tokio-compat` crate to wrap the existing tokio 0.2
runtime, rather than constructing its own runtime.
Because the runtime does not expose an `around_worker` API, it cannot
currently be used with the 0.1 `with_default` method of setting the
reactor, timer, and executor. This means that tokio-compat must
duplicate a lot of existing code from `tokio` to construct the runtime,
which is unfortunate (and has the potential to introduce errors). On the
other hand, we can use the drop guard APIs with `before_start` and
`after_stop`, by storing the drop guards in a thread-local. This will
allow `tokio-compat` to wrap the 0.2 runtime, reducing code duplication.
Also, this will allow the blocking pool to be used on the compat
runtime, which is currently impossible (as the blocking APIs are private
to `tokio`).
Signed-off-by: Eliza Weisman <[email protected]>
## Motivation
Currently, the tests for `tokio` 0.1's async-await support build against a
fairly old nightly from Rust 1.36. Upstream changes to a transitive
dependency introduced a use of `MaybeUninit`, which is feature
flagged on this nightly. This resultedin [0.1.x builds breaking][1].
The `tokio` 0.1 async-await support has not been maintained, in favour
of working on 0.2. It currently uses severely outdated versions of the
async-await APIs (including the `await!` macro). Anyone using
async-await with Tokio is almsot certainly on 0.2 by now.
## Solution
Since the 0.1 async-await APIs are both unused and unmaintained, this
branch deletes them.
[1]: https://dev.azure.com/tokio-rs/Tokio/_build/results?buildId=3174&view=logs&jobId=ba363064-0d45-526e-6c63-c7e816804fbe&taskId=3aff0ee6-e312-56d4-f5d3-d804e6c343c3&lineStart=83&lineEnd=87&colStart=1&colEnd=1
Signed-off-by: Eliza Weisman <[email protected]>
When polling the task, the current waker is saved to the oneshot state.
When the handle is migrated to a new task and polled again, the waker
must be swaped from the old waker to the new waker. In some cases, there
is a potential for the old waker to leak.
This bug was caught by loom with the recently added memory leak
detection.
Backport of #1648.
* upgrade to rand 0.7.0 (MSRV 1.32)
* upgrade to parking_lot 0.9.0
* Remove last non-dev dependency on rand crate (#1324)
Use std RandomState for XorShift seeding. This allows dropping _rand_
crate dep here, accept as a dev dependency for tests or benchmarks.
* increase CI MSRV to 1.31.0
* increase nightly for CI TSAN tests
* add TSAN suppressions for recent rand related updates
* make latest TSAN suppression patterns more general
* upgrade tempfile dev dep for common rand version
But avoid tempfile 3.2 for now, since history demonstrates it bumps
rand versions and MSRV in MINOR updates.
* update (dev dep) env_logger to latest 0.6
* reactor, threadpool: bump PATCH versions, doc links, change logs [ci-release]
* use dyn Trait syntax where appropriate
recent rust nightly started warning that not using `dyn` was
deprecated. This requires MSRV 1.27.0+.
* rustfmt fallout from dyn additions
* stop explicit allow of rust_2018_idioms
* more dyn Trait syntax
* drop tokio-macros from 0.1.x workspace
Since tokio-macros specifies an edition=2018, we would otherwise
require MSRV 1.31.0 to build/test it. And tokio-macros isn't used with
tokio 0.1.x.
* reactor: narrow tokio-io-pool dev dep to 0.1.4
Since 0.1.5-6 is now a edition=2018 crate, which has effective MSRV
1.31.0.
* narrow tempfile dev-dep to avoid MSRV bump
tempfile 3.1.0 pulls in rand 0.7.0 and is MSRV 1.32.0
* narrow flate2 dev-dep to avoid MSRV bump
flate2 1.0.10-11 have MSRV 1.34.0.
github refs: alexcrichton/flate2-rs#207
* fs: drop deprecated tempdir crate use in tests
In particular because it pulls in old rand duplicates. Replace use
with tempfile::tempdir() which has been available since tempfile
3.0.0.
backport-of: #1312
* increase CI MSRV to 1.28.0
* chore: remove `tokio-trace`, add "Related Projects" to README (#1221)
The `tokio-trace` and `tokio-trace-core` crates have been renamed to
`tracing` and `tracing-core`, and moved to their own repository
(`tokio-rs/tracing`).
This branch removes `tokio-trace` and `tokio-trace-core` from the
`tokio` repository. In addition, I've added a "Related Projects" section
to the root README, which lists `tracing` (as well as `mio`, and
`bytes`) as other libraries maintained by the Tokio project. I thought
that this would help folks looking for `tokio-trace` here find it in its
new home.
In addition, it changes `tokio` to depend on `tracing-core` rather than
`tokio-trace-core`.
Closes#1159
Signed-off-by: Eliza Weisman <[email protected]>
* Remove erroneous add of `tokio-macros`
* Some more tokio-trace remnants
* Remove tokio-trace remnants from Cirrus CI
* disable tracing-core feature by default
It can't build on Rust 1.26.0
Signed-off-by: Eliza Weisman <[email protected]>
* fix feature flagging
Signed-off-by: Eliza Weisman <[email protected]>
Callers may not always have `futures` available at the root of the
crate. Re-exporting dependencies makes them available to the macro at a
deterministic location.
The signal-hook library got split into lower-level and higher-level
parts. The tokio-signal uses only API from the lower-level one, so it
can depend on it directly.
The only effect of this change is smaller amount of compiled (and
unused) code during compilation. There's no change in the code actually
used.
This PR introduces `Lock`: A concurrency primitive built on top of `Semaphore` that provides a `Mutex`-like primitive that interacts nicely with futures. Specifically, `LockGuard` (in contrast to `MutexGuard`) does _not_ borrow the `Lock`, and can thus be passed into a future where it will later be unlocked.
This replaces #958, which attempted to introduce a less generic version. The primitive proposed there will instead live in [`async-lease`](https://github.com/jonhoo/async-lease).
This branch changes `dispatcher::get_default` to unset the thread's
current dispatcher while the reference to it is held by the closure.
This prevents infinite loops if the subscriber calls code paths which
emit events or construct spans.
Note that this also means that nested calls to `get_default` inside of a
`get_default` closure will receive a `None` dispatcher rather than the
"actual" dispatcher. However, it was necessary to unset the default in
`get_default` rather than in dispatch methods such as `Dispatch::enter`,
as when those functions are called, the current state has already been
borrowed.
Before:
```
test enter_span ... bench: 3 ns/iter (+/- 0)
test span_no_fields ... bench: 51 ns/iter (+/- 12)
test span_repeatedly ... bench: 5,073 ns/iter (+/- 1,528)
test span_with_fields ... bench: 56 ns/iter (+/- 49)
test span_with_fields_record ... bench: 363 ns/iter (+/- 61)
```
After:
```
test enter_span ... bench: 3 ns/iter (+/- 0)
test span_no_fields ... bench: 35 ns/iter (+/- 12)
test span_repeatedly ... bench: 4,165 ns/iter (+/- 298)
test span_with_fields ... bench: 48 ns/iter (+/- 12)
test span_with_fields_record ... bench: 363 ns/iter (+/- 91)
```
Closes#1032
Signed-off-by: Eliza Weisman <[email protected]>
* Fix crate path in `Cargo.toml` of examples
* Add `edition2018` to examples in the documentation to make it compiled
on Rust 2018
* Fix an example in the documentation
## Motivation
To ease the implementation of `Subscriber::register_callsite`, a field
should be added to `Metadata` to indicate if this callsite is an event or
a span.
## Solution
A new struct, `Kind`, is added to the `metadata` module in
`tokio-trace-core`, and a `Kind` field is added to the `Metadata`
struct. Macros which construct `metadata` now require a `Kind`.
`Kind` is represented as a struct with a private inner enum to allow new
`Kind`s to be added without breaking changes. However, the _addition_ of
the kind field _is_ a breaking change. While this could be done in a
backward-compatible way, it would permit the construction of metadata
with unknown kinds, and since the next `tokio-trace-core` release will
be a breaking change, I opted to make the breaking change instead.
New API tests for the `callsite!` and `metadata!` macros have been added
to guard against future API breakage.
Fixes: #986Closes: #1008
Co-Authored-By: csmoe <[email protected]>
## Motivation
Currently, `tokio-trace-core` permits `Subscriber`s to indicate that
they are "always", "sometimes", or "never" interested in a particular
callsite. When "always" or "never" is returned, then the interest is
cached and the subscriber will not be asked again about that callsite.
This is much more efficient than requiring the filter to be re-evaluated
every time the callsite is hit.
However, if a subscriber wishes to change its filter configuration
dynamically at runtime, it cannot benefit from this caching. Instead, it
must always return `Interest::sometimes`. Even when filters change very
infrequently, they must still always be re-evaluated every time.
In order to support a use-case where subscribers may change their filter
configuration at runtime (e.g. tokio-rs/tokio-trace-nursery#42),
but do so infrequently, we should introducing a new function to
invalidate the cached interest.
## Solution
This branch adds a new function in the `callsite` module, called
`rebuild_interest_cache`, that will invalidate and rebuild all cached
interest.
## Breaking Change
In order to fix a race condition that could occur when rebuilding
interest caches using `clear_interest` and `add_interest`, these methods
have been replaced by a new `set_interest` method. `set_interest` should
have the semantics of atomically replacing the previous cached interest,
so that the callsite does not enter a temporary state where it has no
interest.
Closes#1038
Co-Authored-By: yaahallo <[email protected]>
## Motivation
The `Span::enter` function previously required an `&mut` reference to
enter a span. This is a relic of an earlier design where span closure
logic was determined by dropping an inner span component, and is no
longer strictly necessary.
Requiring `&mut self` to enter a span leads to awkward patterns in cases
when a user wishes to enter a span and then call methods on the span
(such as recording field values). For example, we cannot say
```rust
let mut span = span!("foo", bar);
span.enter(|| {
span.record("bar" &false);
});
```
since the span is mutably borrowed by `enter`. Instead, we must clone
the span, like so:
```rust
let mut span = span!("foo", bar);
span.clone().enter(|| {
span.record("bar" &false);
});
```
Having to clone the span is somewhat less ergonomic, and it has
performance disadvantages as well: cloning a `Span` will clone the
span's `Dispatch` handle, requiring an `Arc` bump, as well as calling
the `Subscriber`'s `clone_span` and `drop_span` functions. If we can
enter spans without a mutable borrow, we don't have to update any of
these ref counts.
The other reason we may wish to require mutable borrows to enter a span
is if we want to disallow entering a span multiple times before exiting
it. However, it is trivially possible to re-enter a span on the same
thread regardless, by cloning the span and entering it twice. Besides,
there may be a valuable semantic meaning in entering a span from inside
itself, such as when a function is called recursively, so disallowing
this is not a goal.
## Solution
This branch rewrites the `Span::enter`, `Span::record`, and
`Span::record_all` functions to no longer require mutable borrows.
In the case of `record` and `record_all`, this was trivial, as borrowing
mutably was not actually *necessary* for those functions. For `enter`,
the `Entered` guard type was reworked to consist of an `&'a Inner`
rather than an `Inner`, so it is no longer necessary to `take` the
span's `Inner`.
## Notes
In addition to allowing spans to be entered without mutable borrows,
`Entered` was changed to exit the span automatically when the guard is
dropped, so we may now observe correct span exits even when unwinding.
Furthermore, this allows us to simplify the `enter` function a bit,
leading to a minor performance improvement when entering spans.
Before:
```
test enter_span ... bench: 13 ns/iter (+/- 1)
```
...and after:
```
test enter_span ... bench: 3 ns/iter (+/- 1)
```
Note that this branch also contains a change to make the
`subscriber::enter_span` benchmark more accurate. Previously, this
benchmark constructed a new span inside of `b.iter(|| {...})`. This
means that the benchmark was measuring not only the time taken to enter
a span, but the time taken to construct a `Span` handle as well.
However, we already have benchmarks for span construction, and the
intention of this particular benchmark was to measure the overhead of
constructing a span.
I've updated the benchmark by moving the span construction out of the
`iter` closure. Now, the span is constructed a single time and entered
on every iteration. This allows us to measure only the overhead of
actually entering a span. The "before" benchmark numbers above were
recorded after backporting this change to master, so they are "fair" to
the previous implementation. Prior to this change the benchmark took
approximately 53 ns.
Signed-off-by: Eliza Weisman <[email protected]>
## Motivation
In order to support conventions that add namespacing to `tokio-trace`
field names, it's necessary to accept at least one type of separator
character. Currently, the `tokio-trace` macros only accept valid Rust
identifiers, so there is no clear separator character for namespaced
conventions. See also #1018.
## Solution
This branch changes the single `ident` fragment matcher for field names
to match *one or more* `ident` fragments separated by `.` characters.
## Notes
The resulting key is still exposed to `tokio-trace-core` as a string
constant created by stringifying the dotted expression. However, if
`tokio-trace-core` were later to adopt a first class notion of
hierarchical field keys, we would be able to track that change in
`tokio-trace` as an implementation detail.
Closes#1018.
Closes#1022.
Signed-off-by: Eliza Weisman <[email protected]>
This branch fixes the `tokio-trace` Subscriber benchmarks panicking due
to constructing spans with ID 0. They will now use an arbitrary constant
instead.
Signed-off-by: Eliza Weisman <[email protected]>
## Motivation
Was determined that having the span! macro default to the TRACE level is
probably not ideal (see discussion on #952).
Closes#1013
## Solution
Remove default trace level and make log lvl mandatory on span! macro,
and add the respective `trace_span!`, `debug_span!`, `info_span!`,
`warn_span!` and `error_span!` macros that behave as span! macro, but
with defined log levels
## Notes
I think this is it, also removed some captures that were repeated, and
some testcases that also seemed repeated after adding the mandatory log
level, but please review it, if more tests or examples are needed happy
to provide (tried to find a way to get the generated macros log level,
but didn't find one, if there is a way i can add tests to assert that
the generated macro has the matching log level ). thanks
- Use `Handle::default` over `Handle::current` for consistent semantics
- Make all `windows::Event` constructors lazily invoke `global_init`
so they can be safely constructed off-task
- Don't assume the reactor is alive and event registration will be done
when calling `global_init`
Add windows regression tests. Unfortunately, Windows doesn't have a
reliable way of programmatically sending CTRL_C or CTRL_BREAK events
to a progress, so the tests can only exercise our internal machinery by
invoking the handler that we register with the OS
Fixes#999
This branch modifies the `tokio_trace::Span` API functions that take
span IDs (the `Span::child_of` constructor, and the `Span::follows_from`
method) so that more types bearing a span ID can be passed as an
argument. Span IDs may now be passed directly without requiring them to
be passed as `Some(id)`. This should make the API slightly more
ergonomic.
Also, it changes the `Span::field` method to take an `AsField` rather
than a `Borrow<str>`.
Signed-off-by: Eliza Weisman <[email protected]>
This branch improves the `fmt::Debug` implementation for `Metadata`,
and adds `fmt::Display` implementations for `FieldSet` and `ValueSet`.
When formatting a `Metadata`, only present fields are formatted --- if
optional fields, such as the file, line number, and module path are
`None`, they will be excluded. In addition, `Metadata` now formats its
`FieldSet` using `FieldSet`'s `fmt::Display` implementation, which is a
bit less noisy. Finally, the `Debug` output for `Metadata` now includes
the callsite that the metadata originates from.
The intention behind these changes is to make the output from failed
tests somewhat easier to interpret.
Signed-off-by: Eliza Weisman <[email protected]>
## Motivation
`tokio-trace` currently offers a strategy for compatibility with the
`log` crate: its macros can be dropped in as a replacement for `log`'s
macros, and a subscriber can be used that translates trace events to log
records. However, this requires the application to be aware of
`tokio-trace` and manually set up this subscriber.
Many libraries currently emit `log` records, and would like to be able
to emit `tokio-trace` instrumentation instead. The `tokio` runtimes are
one such example. However, with the current log compatibility strategy,
replacing existing logging with trace instrumentation would break
`tokio`'s logs for any downstream user which is using only `log` and not
`tokio-trace`. It is desirable for libraries to have the option to emit
both `log` _and_ `tokio-trace` diagnostics from the same instrumentation
points.
## Solution
This branch adds a `log` feature flag to the `tokio-trace` crate, which
when set, causes `tokio-trace` instrumentation to emit log records as well
as `tokio-trace` instrumentation.
## Notes
In order to allow spans to log their names when they are entered and
exited even when the span is disabled, this branch adds an
`&'static Metadata` to the `Span` type. This was previously stored in
the `Inner` type and was thus only present when the span was enabled.
This makes disabled spans one word longer, but enabled spans remain
the same size.
Fixes: #949
Signed-off-by: Eliza Weisman <[email protected]>
## Motivation
`tokio-trace` should have static verbosity level filtering, like the
`log` crate. The static max verbosity level should be controlled at
compile time with a set of features. It should be possible to set a
separate max level for release and debug mode builds.
## Solution
We can do this fairly similarly to how the `log` crate does it:
`tokio-trace` should export a constant whose value is set based on the
static max level feature flags. Then, we add an if statement to the
`span!` and `event!` macros which tests if that event or span's level
is enabled.
Closes#959
## Motivation
In order to implement "out of band" `Subscriber` APIs in third-party
subscriber implementations (see [this comment]) users may want to
downcast the current `Dispatch` to a concrete subscriber type.
For example, in a library for integrating `tokio-trace` with a fancy new
(hypothetical) distributed tracing technology "ElizaTracing", which uses
256-bit span IDs, we might expect to see a function like this:
```rust
pub fn correlate(tt: tokio_trace::span::Id, et: elizatracing::SpanId) {
tokio_trace::dispatcher::with(|c| {
if let Some(s) = c.downcast_ref::<elizatracing::Subscriber>() {
s.do_elizatracing_correlation_magic(tt, et);
}
});
}
```
This allows users to correlate `tokio-trace` IDs with IDs in the
distributed tracing system without having to pass a special handle to
the subscriber through application code (as one is already present in
thread-local storage, but with its type erased).
## Solution
This branch makes the following changes:
* Add an object-safe `downcast_raw` method to the `Subscriber` trait,
taking a `TypeId` and returning an `*const ()` if the type ID
matches the subscriber's type ID, or `None` if it does not, and
* Add `is<T>` and `downcast_ref<T>` functions to `Subscriber`
and `Dispatch`, using `downcast_raw`.
Unlike the approach implemented in #950, the `downcast_raw` method is
object-safe, since it takes a `TypeId` rather than a type _parameter_
and returns a void pointer rather than an `&T`. This means that
`Subscriber` implementations can override this method if necessary. For
example, a `Subscriber` that fans out to multiple component subscribers
can downcast to their component parts, and "chained" or "middleware"
subscribers, which wrap an inner `Subscriber` and modify its behaviour
somehow, can downcast to the inner type if they choose to.
[this comment]: https://github.com/tokio-rs/tokio/issues/932#issuecomment-469473501
[`std::error::Error`'s]: https://doc.rust-lang.org/1.33.0/src/std/error.rs.html#204
Refs: #950, #953, https://github.com/tokio-rs/tokio/issues/948#issuecomment-469444293
Signed-off-by: Eliza Weisman <[email protected]>
* chore: Fix examples not working with `cargo run`
## Motivation
PR #991 moved the `tokio` crate to its own subdirectory, but did not
move the `examples` directory into `tokio/examples`. While attempting to
use the examples for testing another change, I noticed that #991 had
broken the ability to use `cargo run`, as the examples were no longer
considered part of a crate that cargo was aware of:
```
tokio on master [$] via 🦀v1.33.0 at ☸️ aks-eliza-dev
➜ cargo run --example chat
error: no example target named `chat`
Did you mean `echo`?
```
## Solution
This branch moves the examples into the `tokio` directory, so cargo is
now once again aware of them:
```
tokio on eliza/fix-examples [$] via 🦀v1.33.0 at ☸️ aks-eliza-dev
➜ cargo run --example chat
Compiling tokio-executor v0.1.7 (/Users/eliza/Code/tokio/tokio-executor)
Compiling tokio-reactor v0.1.9
Compiling tokio-threadpool v0.1.13
Compiling tokio-current-thread v0.1.6
Compiling tokio-timer v0.2.10
Compiling tokio-uds v0.2.5
Compiling tokio-udp v0.1.3
Compiling tokio-tcp v0.1.3
Compiling tokio-fs v0.1.6
Compiling tokio v0.1.18 (/Users/eliza/Code/tokio/tokio)
Finished dev [unoptimized + debuginfo] target(s) in 7.04s
Running `target/debug/examples/chat`
server running on localhost:6142
```
Signed-off-by: Eliza Weisman <[email protected]>
Signed-off-by: Eliza Weisman <[email protected]>
#993 introduces changes in a sub crate that other Tokio crates depend
on. To make CI pass, a `[patch]` statement and `path` dependencies are
used.
When releasing, these must be removed. However, the commit that
removes them and prepares the crates for release will not be able to
pass CI.
This commit adds a conditional on a special `[ci-release]` snippet in
the commit message. If this exists, CI is only run with the full "patched"
dependencies.
Adds a `TypedExecutor` trait that describes how to spawn futures of a specific
type. This is useful for implementing functions that are generic over an executor
and wish to support both `Send` and `!Send` cases.
This branch makes the following changes to `tokio-trace`'s `Span` type:
* **Remove manual close API from spans**
In practice, there wasn't really a use-case for this, and it
complicates the implementation a bit. We can always add it back later.
* **Remove generic lifetime from `Span`**
Again, there wasn't actually a use-case for spans with metadata that
doesn't live for the static lifetime, and it made using `Span`s in
other types somewhat inconvenient. It's also possible to implement an
alternative API for non-static spans on top of the `tokio-trace-core`
primitives.
Signed-off-by: Eliza Weisman <[email protected]>
PR #973 changed the `tokio_trace_core::span::Id::from_u64` function to
require that the provided `u64` be greater than zero. However, I had
forgotten that the implementation of `Subscriber` for the `NoSubscriber`
type (which is used when no default subscriber is set) always returned
`span::Id::from_u64(0)` from its `new_span` method. In combination with
the assert added in #973, this means that every time a span is hit when
no subscriber is set, `tokio-trace-core` will panic.
This branch fixes the panics by having `NoSubscriber` construct span IDs
using a different (arbitrarily chosen) non-zero constant.
Signed-off-by: Eliza Weisman <[email protected]>
This branch changes `tokio_trace_core::span::Id::from_u64` to assert
that the integer from which the span ID is constructed is greater than
zero. This is to enable future use of non-zero optimization.
Unfortunately, we can't actually use a `NonZeroU64` _now_, as that type
was only stabilized in Rust 1.28.0, and `tokio`'s current minimum
supported Rust version is 1.26.0.
Adding and documenting the assertion now allows us to change the
internal representation to `NonZeroU64` later (when 1.28.0 is the
minimum supported Rust version), without causing a breaking change.
Signed-off-by: Eliza Weisman <[email protected]>
* trace-core: Pass dispatcher by ref to `dispatcher::with_default`
As requested by @carllerche in https://github.com/tokio-rs/tokio/pull/966#discussion_r264380005, this branch changes the
`dispatcher::with_default` function in `tokio-trace-core` to take the
dispatcher by ref and perform the clone internally. This makes this
function more consistant with other `with_default` functions in other
crates.
Signed-off-by: Eliza Weisman <[email protected]>
* trace: Don't set the default dispatcher on entering a span
Setting the default dispatcher on span entry is a relic of when spans
tracked their parent's ID. At that time, it was necessary to ensure that
any spans created inside a span were observed by the same subscriber
that originally provided the entered span with an ID, as otherwise, new
spans would be created with parent IDs that did not originate from that
subscriber.
Now that spans don't track their parent ID, this is no longer necessary.
However, removing this behavior does mean that if a span is entered
outside of the subscriber context it was created in, any subsequent
spans will be observed by the current default subscriber and thus will
not be part of the original span's trace tree. Since subscribers are not
expected to change frequently, and spans are not expected to move
between them, this is likely acceptable.
I've removed the tests for the old behavior.
Note that this change improves the performance of span entry/exit fairly
significantly. Here are the results of running a benchmark that enters
a span, does nothing, and immediately exits it, before this change:
```
test enter_span ... bench: 93 ns/iter (+/- 14)
```
...and after:
```
test enter_span ... bench: 51 ns/iter (+/- 9)
```
Signed-off-by: Eliza Weisman <[email protected]>
## Motivation
Currently, it isn't possible to import individual macros from
`tokio-trace` using the macros 1.2 syntax:
```rust
use tokio_trace::{debug, info, span};
```
This is because these macros require that `callsite` and `enabled` are
imported as well.
## Solution
This branch resolves the problem by adding the [`local_inner_macros`]
attribute to the instrumentation API's macros. This allows other macros
from within the crate to be used without requiring them to be explicitly
imported.
However, this also requires duplicating any macros from other sources
(such as std and `tokio-trace-core`) with wrappers due to the behaviour
of `local_inner_macros`. I've added these wrapper macros as well.
Since the macros got even longer as a result of this, I've moved them
to a separate file to make `lib.rs` easier to read. I've also wrapped
some very long lines in the macros, and removed the explicit drop of
the result of evaluating some event macros (it's no longer necessary
as all event macros now evaluate to `()`).
[`local_inner_macros`]: https://doc.rust-lang.org/nightly/edition-guide/rust-2018/macros/macro-changes.html#local-helper-macrosFixes#968
Signed-off-by: Eliza Weisman <[email protected]>
This branch makes a handful of `tokio-trace-core` API improvements, mostly
around naming. In particular:
* Rename `dispatcher::with` to `dispatcher::get_default`
* Rename `Event::observe` to `Event::dispatch`
* Make `field::ValidLen` trait private
Closes#948Closes#960
Signed-off-by: Eliza Weisman <[email protected]>
This branch changes the `Subscriber::record` method to take a new
arguments struct, `span::Record`. The `field::Record` trait was renamed
to `field::Visit` to prevent name conflicts.
In addition, the `ValueSet::is_empty`, `ValueSet::contains`, and
`ValueSet::record` methods were made crate-private, as they are exposed
on the `Attributes` and `Record` types.
Signed-off-by: Eliza Weisman <[email protected]>
This branch allows users of `tokio-trace` to explicitly set a span's
parent, or indicate that a span should be a new root of its own trace
tree. A `parent: ` key has been added to the `span!` macros. When a span
is provided, that span will be set as the parent, while `parent: None`
will result in a new root span. No `parent:` key results in the current
behaviour.
A new type, `span::Attributes`, was added to `tokio-trace-core` to act
as an arguments struct for the `Subscriber::new_span` method. This will
allow future fields to be added without causing breaking API changes.
The `Attributes` struct currently contains the new span's metadata,
`ValueSet`, and parent.
Finally, the `span::Span` type in `-core` was renamed to `span::Id`, for
consistency with `tokio-trace` and to differentiate it from
`span::Attributes`. This name was chosen primarily due to precedent in
other tracing systems.
Closes#920
Signed-off-by: Eliza Weisman <[email protected]>
A single-producer, multi-consumer channel that only retains the _last_ sent
value. Values are broadcasted out.
This channel is useful for watching for changes to a value from multiple
points in the code base (for example, changes to a configuration value).
- Rewrite noop_waker with items from the new API and replaces
LocalWaker with Waker.
- Bump the minimum required version for `tokio-async-await` to
1.34.0-nightly.
- `Unpin` was added to std prelude.
- Add `cargo check` to .travis.yml
Fixes: #908
This branch adds links to the master RustDoc published by CI to the
`tokio-trace` and `tokio-trace-core` README. In addition, it fixes a
broken links in the RustDoc for `tokio-trace` and updates the
`tokio-trace-core` RustDoc to match the README.
Signed-off-by: Eliza Weisman <[email protected]>
## Motivation
As described in #905, subscribers have no way to get the numeric value of a span ID back _out_ of a `Span`.
## Solution
Add a `Span::into_u64` method that returns the inner `u64` span ID
Closes#905
Signed-off-by: kleimkuhler <[email protected]>
In order to support nesting a tokio reactor within another event
system exposing the file descriptor for the underlying reactor
is useful and is already implemented for mio::Poll.
Signed-off-by: Paul Osborne <[email protected]>
This patch fixes Semaphore by adding a missing code path to the release
routine that handles the case where the waiter's node is queued in the
sempahore but has not yet been assigned the permit.
This fix is used by mpsc to handle the case when the Sender has called
`poll_ready` and is dropped before the permit is acquired.
Fixes#900
This change adds an extension trait to `BufStream` and puts the core
trait behind a feature flag for optional use.
This mainly adds the additional functions in an extension trait to
allow the user to select if they want just the core trait or the fully
featured version. Now the user can add the core feature to _not_
include the extension trait. By deafult, this feature is disabled.
<!-- Thank you for your Pull Request. Please provide a description above
and review the requirements below.
Bug fixes and new features should include tests.
Contributors guide:
https://github.com/tokio-rs/tokio/blob/master/CONTRIBUTING.md -->
## Motivation
In asynchronous systems like Tokio, interpreting traditional log
messages can often be quite challenging. Since individual tasks are
multiplexed on the same thread, associated events and log lines are
intermixed making it difficult to trace the logic flow. Currently, none
of the available logging frameworks or libraries in Rust offer the
ability to trace logical paths through a futures-based program.
There also are complementary goals that can be accomplished with such a
system. For example, metrics / instrumentation can be tracked by
observing emitted events, or trace data can be exported to a distributed
tracing or event processing system.
In addition, it can often be useful to generate this diagnostic data in
a structured manner that can be consumed programmatically. While prior
art for structured logging in Rust exists, it is not currently
standardized, and is not "Tokio-friendly".
## Solution
This branch adds a new library to the tokio project, `tokio-trace`.
`tokio-trace` expands upon logging-style diagnostics by allowing
libraries and applications to record structured events with additional
information about *temporality* and *causality* --- unlike a log
message, a span in `tokio-trace` has a beginning and end time, may be
entered and exited by the flow of execution, and may exist within a
nested tree of similar spans. In addition, `tokio-trace` spans are
*structured*, with the ability to record typed data as well as textual
messages.
The `tokio-trace-core` crate contains the core primitives for this
system, which are expected to remain stable, while `tokio-trace` crate
provides a more "batteries-included" API. In particular, it provides
macros which are a superset of the `log` crate's `error!`, `warn!`,
`info!`, `debug!`, and `trace!` macros, allowing users to begin the
process of adopting `tokio-trace` by performing a drop-in replacement.
## Notes
Work on this project had previously been carried out in the
[tokio-trace-prototype] repository. In addition to the `tokio-trace` and
`tokio-trace-core` crates, the `tokio-trace-prototype` repo also
contains prototypes or sketches of adapter, compatibility, and utility
crates which provide useful functionality for `tokio-trace`, but these
crates are not yet ready for a release. When this branch is merged, that
repository will be archived, and the remaining unstable crates will be
moved to a new `tokio-trace-nursery` repository. Remaining issues on the
`tokio-trace-prototype` repo will be moved to the appropriate new repo.
The crates added in this branch are not _identical_ to the current head
of the `tokio-trace-prototype` repo, as I did some final clean-up and docs
polish in this branch prior to merging this PR.
[tokio-trace-prototype]: https://github.com/hawkw/tokio-trace-prototypeCloses: #561
Signed-off-by: Eliza Weisman <[email protected]>
ATOMIC_BOOL_INIT is deprecated since 1.34 because the const fn
AtomicUsize::new is now preferred. As deny(warnings) is set,
tokio fails to build on latest nightly. This will fix it.
Signed-off-by: Yilin Chen <[email protected]>
Following from https://github.com/tokio-rs/tokio/pull/865, this PR
removes `#[derive(Debug)]` on `mpsc` sender and receiver types in favor
of explicit `impl fmt::Debug` blocks that don't have a `T: fmt::Debug`
bound.
`#[derive(Clone)]` on a type `struct Foo<T>` adds an impl that requires that
`T: Clone`:
```rust
impl<T: Clone> Clone for Foo<T>
```
which is unfortunate in the case of senders, because we don't want to require
that the items being sent are `Clone` for the channel sender to be `Clone`.
This PR adds an explicit `impl Clone` for the bounded and unbounded sender
types which does not have the `T: Clone` bound.
Note that this is _also_ an issue with `#[derive(Debug)]`, but that one is
harder to work around as `chan::Tx` _also_ has `#[derive(Debug)]`, as does
`chan::Chan`, so we'd have to add explicit impls for all of them to make
progress.
Introduce a tokio-sync crate containing useful synchronization primitives for programs
written using Tokio.
The initial release contains:
* An mpsc channel
* A oneshot channel
* A semaphore implementation
* An `AtomicTask` primitive.
The `oneshot` and `mpsc` channels are new implementations providing improved
performance characteristics. In some benchmarks, the new mpsc channel shows
up to 7x improvement over the version provided by the `futures` crate. Unfortunately,
the `oneshot` implementation only provides a slight performance improvement as it
is mostly limited by the `futures` 0.1 task system. Once updated to the `std` version
of `Future` (currently nightly only), much greater performance improvements should
be achievable by `oneshot`.
Additionally, he implementations provided here are checked using
[Loom](http://github.com/carllerche/loom/), which provides greater confidence of
correctness.
## Motivation
When the thread pool shuts down, futures that have been polled at least once but not completed yet are simply leaked. We should drop them instead.
## Solution
Multiple changes are introduced:
* Tasks are assigned a home worker the first time they are polled.
* Each worker contains a set of tasks (`Arc<Task>`) it is home to. When a task is assigned a home worker, it is registered in that worker's set of tasks. When the task is completed, it is unregistered from the set.
* When the thread pool shuts down and after all worker threads stop, the remaining tasks in workers' sets are aborted, i.e. they are switched to the `Aborted` state and their `Future`s are dropped.
* The thread pool shutdown process is refactored to make it more robust. We don't track the number of active threads manually anymore. Instead, there's `Arc<ShutdownTrigger>` that aborts remaining tasks and completes the `Shutdown` future once it gets destroyed (when all `Worker`s and `ThreadPool` get dropped because they're the only ones to contain strong references to the `ShutdownTrigger`).
Closes#424Closes#428
* docs: fixed links to tokio_timer::clock::Now in tokio-timer/src/timer/mod.rs
* docs: fixed links to std::time::Instant in tokio-timer/src/timer/mod.rs
This reverts commit 7a49ebb65e.
The commit conflicted with another change that was merged, causing CI to fail. The public API
also requires a bit more refinement (#833) and Tokio crates need to be released.
Disabling all features means the only dependency is `futures`.
Relevant pieces of the API can then be enabled with the following features:
- `codec`
- `fs`
- `io`
- `reactor`
- `tcp`
- `timer`
- `udp`
- `uds`
This also introduces the beginnings of enabling only certain pieces of the `Runtime`. As a start, the entire default runtime API is enabled via the `rt-full` feature.
Error::cause is deprecated in Rust 1.33, but this allows Error::cause
until the minimum supported version of tokio is Rust 1.30.
When the minimum support version of tokio reaches Rust 1.30,
replace Error::cause with Error::source.
Fixes: #817
`inner` is a fitting name for variables of type named `Inner`, but in other cases I find them confusing - sometimes `inner` refers to a `Pool`, sometimes to a `Sender`. I renamed a bunch of variables named `inner` to be more descriptive.
This PR is the first step in an effort of splitting https://github.com/tokio-rs/tokio/pull/722#issuecomment-439552671 into multiple PRs.
When spawning using `Handle` while on the executor, tasks were being
double counted. This prevented the number of active tasks to reach zero,
thus preventing the executor from shutting down.
This changes `spawn` to check if being called from the executor
**before** incrementing the number of active tasks.
Fixes#760
Should hopefully fix the underlying bug that was causing tokio-tls tests to occasionally fail on Windows.
Signed-off-by: Toby Lawrence <[email protected]>
* async-await: fix README example dependencies
As per commit "async-await: track nightly changes (#661)" ( commit
2f690d30bc)
> The `tokio-async-await` crate is no longer a facade. Instead, the
> `tokio` crate provides a feature flag to enable async/await support.
Ensure the example in the async-await README file also works by
correctly declaring this updated dependency
* async-await: remove unnecessary 'edition' declaration from README
As the "edition" feature was stabilized in rust v1.30 and async-await
specifies that the nightly toolchain must be used, remove the use of the
"edition" feature gate since it is enabled by default.
* Minimize allocation needed for channels
* Use a newtype for signal ids
* We can just cast the raw pointer to a `usize` and still perform a
simple identity check, without incurring any implications of storing a
raw pointer (e.g. previously Signal was !Sync and had an unsafe impl of
Send, and now it is naturally Sync+Send)
* Broadcast with `try_send` instead of `start_send`
The `Stream::start_send` method uses backpressure and schedules the
current task to be notified whenever the channel has additional room,
which means we'll generate a lot of unnecessary wakeups whenever a
channel gets full
By changing to `try_send` and handling any errors, we ensure the
Driver's task won't get woken up when a Signal finally consumes its
notification, since we're coalescing things anyway
* udp: add `into_parts` to `RecvDgram`
If `RecvDgram` can not be driven to completion it may become necessary to get back the `UdpSocket` it contains which is currently not possible.
This adds`into_parts` to get the socket as well as the buffer back. Both methods consume `RecvDgram`.
Note that after the future has completed, `into_parts` must not be used, or else a panic will happen.
## Motivation
tokio depends on an out of date version of crossbeam-utils, which results in multiple versions of that package being linked in binaries which use other popular libraries.
## Solution
Bump the version; there's no API changes and tests still pass.
* rt: fix `Runtime::reactor()` as used by tokio-core
Up until Tokio v0.1.11, the handle returned by `Runtime::reactor()`
pointed to a reactor instance running in a background thread. The thread
was eagerly spawned.
As of v0.1.12, a reactor instance is created per runtime worker thread.
`Runtime::reactor()` was deprecated and updated to point to the reactor
for one of the worker threads.
A problem occurs when attempting to use the reactor before spawning a
task. Worker threads are spawned lazily, which means that the reactor
referenced by `Runtime::reactor()` is not yet running.
This patch changes `Runtime::reactor` back to a dedicated reactor
running on a background thread. However, the background thread is now
spawned lazily when the deprecated function is first called.
Fixes#720
* Fix comment
Co-Authored-By: carllerche <[email protected]>
- `tokio::run` checks Enter before creating a new threadpool and
spawning the main future.
- `Runtime::block_on` now checks Enter
- `Runtime::block_on_all` now checks Enter
<!--
Thank you for your Pull Request. Please provide a description above and review
the requirements below.
Bug fixes and new features should include tests.
Contributors guide: https://github.com/tokio-rs/tokio/blob/master/CONTRIBUTING.md
-->
## Motivation
Now that each worker thread drives its own reactor, reactors have to be driven until the threadpool shuts down. We mustn't use the `keep_alive` setting to shut down a worker thread if it doesn't receive an event from the reactor for a certain duration of time.
<!--
Explain the context and why you're making that change. What is the problem
you're trying to solve? In some cases there is not a problem and this can be
thought of as being the motivation for your change.
-->
## Solution
Just ignore the `keep_alive` setting when parking in `Worker::sleep`.
<!--
Summarize the solution and provide any necessary context needed to understand
the code change.
-->
Fixes: #681
## Motivation
Currently, a potential panic exists in `LengthDelimitedCodec::encode`.
Writing the length field to the `dst` buffer can exceed the buffer
capacity, as `BufMut::put_uint_{le,be}` doesn't reserve more capacity.
## Solution
This branch adds a call to `dst.reserve` to ensure that there's
sufficient remaining buffer capacity to hold the length field and
the frame, prior to writing the length field. Previously, capacity
was only reserved later in the function, when writing the frame
to the buffer, and we never reserved capacity for the length field.
I've also added a test that reproduces the issue. The test panics on
master, but passes after making this change.
Signed-off-by: Eliza Weisman <[email protected]>
* io: ensure ReadHalf/WriteHalf do not return WouldBlock directly
These facades were passing back WouldBlock when the internal BiLock
couldn't be acquired, which does not fit the intended behavior.
Signed-off-by: Toby Lawrence <[email protected]>
* io: pull from the local crate, not crates.io
## Motivation
Currently, there is a potential denial of service vulnerability in the
`lines` codec. Since there is no bound on the buffer that holds data
before it is split into a new line, an attacker could send an unbounded
amount of data without sending a `\n` character.
## Solution
This branch adds a `new_with_max_length` constructor for `LinesCodec`
that configures a limit on the maximum number of bytes per line. When
the limit is reached, the the overly long line will be discarded (in
`max_length`-sized increments until a newline character or the end of the
buffer is reached. It was also necessary to add some special-case logic
to avoid creating an empty line when the length limit is reached at the
character immediately _before_ a `\n` character.
Additionally, this branch adds new tests for this function, including a
test for changing the line limit in-flight.
## Notes
This branch makes the following changes from my original PR with
this change (#590):
- The whole too-long line is discarded at once in the first call to `decode`
that encounters it.
- Only one error is emitted per too-long line.
- Made all the changes requested by @carllerche in
https://github.com/tokio-rs/tokio/pull/590#issuecomment-420735023Fixes: #186
Signed-off-by: Eliza Weisman <[email protected]>
## Motivation
Currently, the `RUST_BACKTRACE` environment variable is set to `1` on
Travis CI builds:
https://github.com/tokio-rs/tokio/blob/0ca973a7ebc5b8a29beac1ccb6c73ef26ddcbf22/.travis.yml#L49
However, it's not set on AppVeyor. This can make debugging
Windows-specific CI failures challenging for developers on other
operating systems.
## Solution
This branch sets `RUST_BACKTRACE=1` on AppVeyor.
Signed-off-by: Eliza Weisman <[email protected]>
Since the CI runs all tests for all tokio crates, it is possible that a
sporadic failure in one crate can mask failures/successes of other
crates' tests.
Using the `--no-fail-fast` flag instructs cargo to run *all* tests
before failing the build. This will allow checking to see if any
relevant test cases still pass even if an unrelated test has failed.
* Don't use tokio-core any more for tests. That one brings tokio from
crates.io instead of the current workspace and two versions of that
don't want to cooperate.
* Guard unix-specific examples on windows.
* Leave CI setup to top-level directory.
* Originally reported in alexcrichton/tokio-process#42
* The root cause appears to be due to two different PollEvented
instances trying to consume readiness events from the same file
descriptor.
* Previously we would simply swallow any `AlreadyExists` errors when
attempting to register the pipe receiver with the event loop. I'm not
sure if this means the PollEvented wrapper wasn't fully registered to
receive events, or maybe there is a potential race condition with how
PollEvented consumes mio readiness events. Using a fresh/duplicate file
descriptor appears to mitigate the issue, however.
* I was also not able to reproduce the issue as an isolated test case so
there is no regression test available within this crate (but we can add
one in tokio-process)
* codec: add new constructor `with_max_length ` to `LinesCodec`
* codec: add security note to docs
Signed-off-by: Eliza Weisman <[email protected]>
* Fix Rust 1.25 compatibility
* codec: Fix incorrect line lengths in tests (and add assertions)
Signed-off-by: Eliza Weisman <[email protected]>
* codec: Fix off-by-one error in lines codec
Signed-off-by: Eliza Weisman <[email protected]>
* codec: Fix call to decode rather than decode_eof in test
Signed-off-by: Eliza Weisman <[email protected]>
* codec: Fix incorrect LinesCodec::decode_max_line_length
This bug was introduced after the fix for the off-by-one error.
Fortunately, the doctests caught it.
Signed-off-by: Eliza Weisman <[email protected]>
* codec: Minor style improvements
Signed-off-by: Eliza Weisman <[email protected]>
* codec: Don't allow LinesCodec length limit to be set after construction
Signed-off-by: Eliza Weisman <[email protected]>
* codec: change LinesCodec to error and discard line when at max length
* codec: Fix build on Rust 1.25
The slice patterns syntax wasn't supported yet in that release.
Signed-off-by: Eliza Weisman <[email protected]>
* codec: Add test for out-of-bounds index when peeking
Signed-off-by: Eliza Weisman <[email protected]>
* codec: Fix out of bounds index
* codec: Fix incomplete comment
Signed-off-by: Eliza Weisman <[email protected]>
* codec: Add test for line decoder buffer underrun
@jonhoo reported a panic in the call to `LocalKey::with`, which occurs
when the reactor is dropped in the middle of TLS teardown. This PR
changes the call to `LocalKey::try_with` and handles the case when the
thread-local value has already been destroyed.
* Added timeouts to all tests that were missing them
- any issue we have will likely result in deadlocks/starvation so its
best if all tests quickly timeout rather than require getting killed
or have the CI timeout itself
* Added a `support` module and put a bunch of helpers there to DRY the
tests
* As observed in alexcrichton/tokio-signal#38, Signal instances can starve based on the order
they are created in, and this ordering appears to be platform/OS
specific
* The crux of the issue is that we woud only *attempt* to broadcast any
pending signals if we successfully read out at least one byte from the
global pipe.
* For reasons unclear to me, the affected Signal instance would get
woken up after the signal handler writes to the global pipe, but it
would immediately hit a WouldBlock error and give up, bypassing the
broadcast attempt (even though the pending flag was correctly set).
- Maybe this has to do with OS specifics with how the bytes are
delivered (or not), or with some complex interaction with tokio and
the pipe registration. It seems fishy since strace logs didn't show
the signal handler pipe write fail either, but I'm all out of ideas
* The fix appears simple: unconditionally attempt to broadcast any
pending signals *any* time a Driver instance is woken up.
* Since we perform an atomic check for each pending signal, we know that
each (coalesced) signal broadcast will happen at most once. If we were
supuriously woken up and no signals were pending, then nothing will be
yielded to any pollers of Signal
* The down side is that since each Signal instance polls a Driver
instance, each poll to Signal will essentially perform N atomic
operations (N = number of signals we support) in an attempt to broadcast
any pending signals.
- However, we can revisit optimizing this better in the future
Fixesalexcrichton/tokio-signal#38
* We introduce a new global structure which keeps track of how many
signal streams have been registered with a given event loop (the event
loop is identified by its OS file descriptor)
* We only attempt to deregister our global evented pipe from any event
loop if and only if we are the last signal that was registered with it
* Currently, whenever a new signal stream is created we attempt to
register a global pipe with the event loop to drive events.
* We also (correctly) swallow any descriptor-already-registered errors
since the same pipe is always used
* However, we currently *deregister* the same global pipe *any time* a
Signal stream is dropped.
* This means that if 2 or more of Signal instances exist simultaneously
(even if listening for different signals) and one of them is dropped,
the remainder will starve (until any new signal is created again).
* Cargo runs each integration-style-test in its own process. Since the
tests use global data structures specific to the process, we should run
them in an isolated manner to avoid having cross-test interactions
* Fixesalexcrichton/tokio-signal#39
This allows tokio-signal to build with `-Z minimal-versions` - see
https://github.com/rust-lang/cargo/issues/5657#issuecomment-401110172
for more details.
Earlier versions depend on log 0.3.1, which itself depends on libc
0.1, which doesn't build on any post-1.0 version of rust.
This text historically was copied verbatim from rust-lang/rust's own README [1]
with the intention of licensing projects the same as rustc's own license, namely
a dual MIT/Apache-2.0 license. The clause about "various BSD-like licenses"
isn't actually correct for almost all projects other than rust-lang/rust and
the wording around "both" was slightly ambiguous.
This commit updates the wording to match more precisely what's in the
standard library [2], namely clarifying that there aren't any BSD-like licenses
in this repository and that the source is licensable under either license, at
your own discretion.
[1]: https://github.com/rust-lang/rust/tree/f0fe716dbcbf2363ab8f929325d32a17e51039d0#license
[2]: https://github.com/rust-lang/rust/blob/f0fe716dbcbf2363ab8f929325d32a17e51039d0/src/libstd/lib.rs#L5-L9
Replace the sequential counting (which might be exhausted) by an address
of an object (in a box, so it doesn't change). This is also a unique, so
it is acceptable ID.
Run multiple loops (both in parallel and sequentially) to make sure
broadcasting to multiple of them works and we work even after the
initial loop has gone away.
Add the driver task, connecting the signal handler wakeups to the
wakeups of of the streams.
It is a prototype-quality code, a lot of cleanups and similar is needed.
Which is just a wrapper around the futures::sync::mpsc. The sender is in
a global registry.
The part that connects the wakeups to the senders in the registry
doesn't yet exist.
Register the signal handler that wakes up someone through a self-pipe.
That someone doesn't yet exist, though.
Some dependencies (nix, lazy_static) added to speed up the prototyping
process. They are likely to be dropped in some future commits.
Some features (eg. preserving the previous signal handlers) are still
missing.
## Motivation
`tokio_threadpool::ThreadPool::spawn` has no return value.
## Solution
Add `ThreadPool::spawn_handle` which calls
`futures::sync::oneshot::spawn` to return a future represents the return
value.
This patch refactors `length_delimited` to be implemented as a `Codec` and
use the default `Framed` wrapper types.
The original implementation did not do this in order to support vectored writes in the
write half. However, this implementation would be more efficient with small frames anyway.
If vectored writes are to be explored in the future, then it should be done holistically.
Signed-off-by: Eliza Weisman <[email protected]>
Previously, every call to `current_thread::Handle::spawn` would go
through a `mpsc` channel. This is unnecessary when the `Handle` is still
on the same thread as the current thread executor. This patch fixes that
by storing the `ThreadId` of the executor when it is created, and then
comparing against that when `Handle::spawn` is called. If the call is
made from the same thread, `spawn_local` is used directly.
Fixes#562.
This patch keeps the primary net types in `tokio::net` and moves
secondary types to a protocol specific submodules.
Primary types are the ones that users are most likely to name (`TcpStream`,
`TcpListener`, `UdpSocket`, ...)
Secondary types are the operation futures.
This patch adds experimental async/await support to Tokio. It does this
by adding feature flags to existing libs only where necessary in order
to add nightly specific code (mostly `Unpin` implementations). It then
provides a new crate: `tokio-async-await` which is a shim layer on top
of `tokio`.
The `tokio-async-await` crate is expected to look exactly like `tokio`
does, but with async / await support. This strategy reduces the amount
of cfg guarding in the main libraries.
This patch also adds `tokio-channel`, which is copied from futures-rs
0.1 and adds the necessary `Unpin` implementations. In general, futures
0.1 is mostly unmaintained, so it will make sense for Tokio to take over
maintainership of key components regardless of async / await support.
Re-export it inside the tokio::runtime::current_thread, as the original
place (tokio::executor::current_thread) is hidden from documentation and
users need some way to spawn non-Send futures.
This also bumps a number of sub crates:
* tokio-executor (0.1.3)
* tokio-io (0.1.8)
* tokio-reactor (0.1.4)
* tokio-threadpool (0.1.6)
* tokio-timer (0.2.6)
* tokio-udp (0.1.2)
This patch introduces `Timeout`. This new type allows setting a timeout
both using a duration and an instant. Given this overlap with
`Deadline`, `Deadline` is deprecated.
In addition to supporting future timeouts, the `Timeout` combinator is
able to provide timeout functionality to streams. It does this by
applying a duration based timeout to each item being yielded.
The main reason for introducing `Timeout` is that a deadline approach
does not work with streams. Since `Timeout` needed to be introduced
anyway, keeping `Deadline` around does not make sense.
* Remove `counted` field on `timer::Entry`.
It turns out that a better indicator of whether or not the number of
active timeouts should be decremented is if the `Entry` has been
associated with a timer. In other words, if `Entry::inner` can be
upgraded, then the count should be decremented on drop.
* timer: Tweak link between `Delay` and the driver
This tweaks the struct layout / details regarding how a `Delay` instance
is linked to a driver (timer instance). Instead of lazily allocating the
`Entry` (node shared between `Delay` and the timer), `Entry` is
allocated immediately when `Delay` is created. This allows using the
entry store data used by `Delay`.
This is in anticipation of further timer improvements that would
otherwise require the size of `Delay` to grow further. Since an
allocation is already made, the idea is to shrink the size of the
`Delay` struct.
This patch adds a `DelayQueue` to tokio_timer. The `DelayQueue` allows
inserting elements as well as specifying a time at which the element
should be returned to the user. This allows handling more complex
timeout situations.
This patch implements `Default` for `tokio_timer::Handle`. It returns a
`Handle` instance that is not bound to a specific timer. Instead, it
will use the timer for the current execution context. This is the same
strategy used by `tokio_reactor::Handle`.
Fixes#547
The `lines_encoder` test is a copy/paste of `bytes_encoder` and not
testing the LinesCodec encoding at all. This updates the test to do
simple validations of the LinesCodec encoding.
The futures 0.2 crate is not intended for widespread usage. Also, the
futures team is exploring the compat shim route.
If futures 0.3 support is added to Tokio 0.1, then a different
integration route will be explored, making the current code unhelpful.
* create_dir
* create_dir_all
* hard_link
* read_dir
* read_link
* remove_dir
* remove_file
* rename
* set_permissions that works with path
* symlink_metadata
* symlink on unix
* symlink_dir on windows
* symlink_file on windows
* Normalize links to docs.rs/CRATE/M.N/...
docs.rs is smart enough to show docs for the latest M.N.P release when
M.N is used in the link. For example:
https://docs.rs/mio/0.6/mio/struct.Poll.html
..will show mio 0.6.14 and later docs. While using the `M.N.*`
(ASTERISK) syntax also works, `M.N` is the more common usage, so
standarize a few existing links to that format.
* Fix missing or malformed rustdoc links
* executor lib rustdoc minor format change
* Promote tokio-threadpool crate level comments to rustdoc
* Replace hidden tokio::executor::thread_pool docs with deprecation note
* Fix typo/simplify util module rustdoc
* Reuse some tokio::executor::thread_pool rustdoc for the crate
Relates to #421
This PR enables backtraces when running tests and disables tsan for the thread pool.
The thread sanitizer was generating too many false positives. Once #329 lands, then it can
be re-enabled.
Extract `tokio::executor::current_thread` to a tokio-current-thread
crate. Deprecated fns stay in the old location. The new crate only
contains thee most recent API.
Currently, the timer uses a `Now` trait to abstract the source of time.
This allows time to be mocked out. However, the current implementation
has a number of limitations as represented by #288 and #296.
The main issues are that `Now` requires `&mut self` which prevents a
value from being easily used in a concurrent environment. Also, when
wanting to write code that is abstract over the source of time, generics
get out of hand.
This patch provides an alternate solution. A new type, `Clock` is
provided which defaults to `Instant::now` as the source of time, but
allows configuring the actual source using a new iteration of the `Now`
trait. This time, `Now` is `Send + Sync + 'static`. Internally, `Clock`
stores the now value in an `Arc<Now>` value, which introduces dynamism
and allows `Clock` values to be cloned and be `Sync`.
Also, the current clock can be set for the current execution context
using the `with_default` pattern.
Because using the `Instant::now` will be the most common case by far, it
is special cased in order to avoid the need to allocate an `Arc` and use
dynamic dispatch.
The thread pool optimizes cases where a task currently running on the
pool spawns a new future. However, the optimization did not factor in
cases where two thread pools interacted.
This patch fixes the optimization and includes a test.
Fixes#342
Currently, not specifying a `Handle` is different than using
`Handle::default()`. This is because `Handle::default()` will
immediately bind to the reactor for the current context vs. not
specifying a `Handle`, which binds to a reactor when it is polled.
This patch changes the `Handle::default()` behavior, bringing it inline
with actual defaults.
`Handle::current()` still immediately binds to the current reactor.
Fixes#307
This patch adds a new crate: tokio-fs. This crate provides a wrapper
around `std` functionality that can only be performed using blocking
operations. This primarily includes filesystem operations, but it also
includes standard input, output, and error access as these streams
cannot be safely switched to non-blocking mode in a portable way.
These wrappers call the `std` functions from within a `blocking`
annotation which allows the runtime to compensate for the fact that the
thread will potentially remain blocked in a system call.
Currently, the state of the futures2 integration is pretty broken. This
patch removes the feature flag, preventing users from trying to use it.
In the future, it can be brought back when the implementation is fixed.
This patch introduces a version of `Runtime` that runs all components on
the current thread. This allows users to spawn futures that do not implement
`Send`.
This ensures that all fd-based futures are put into the queue for the
current tick, if the CurrentThread is parking via the Reactor.
Otherwise, if there are queued up futures already, only those would be
polled in the turn. These futures could then notify others/themselves to
have the queue still non-empty on the next turn. Which then potentially
allows the reactor to never be polled, and thus fd-based futures are
never queued up and polled.
Also return in the Turn return value whether any futures were polled at
all, which allows the caller to know if any work was done at all in this
turn and based on that adjust behavior.
This patch adds a `blocking` to `tokio-threadpool`. This function serves
as a way to annotate sections of code that will perform blocking
operations. This informs the thread pool that an additional thread needs
to be spawned to replace the current thread, which will no longer be
able to process the work queue.
Replaces homegrown Arc with std Arc
Is this safer? Unknown. At least we don't have to maintain an arc
implementation anymore. This will also make it easier to filter out tsan
false positives.
Also split task/mod.rs into multiple files.
This patch renames `Sleep` from tokio-timer and the tokio facade to
`Delay`. Given that the future does not actually put anything to sleep,
the `Delay` name feels more appropriate.
Fixes#263
This patch integrate the new timer implementation with the runtime by
initializing a timer per worker thread. This allows minimizing the
amount of synchronization needed for using timers.
* Allow customizing the threadpool's parker
This patch allows the user of threadpool to customize how the worker
threads park themselves. This allows custom parking logic to be
injected. For example, this allows embedding a timer on each worker
thread.
* Call `park` instance every so often.
Since the `park` is now customizable, it might have logic that must be
called every so often. For example, a timer might have timeouts that it
must expire.
Currently, if a worker is very busy, it won't call into the `park`
instance. This patch changes this so that after every 32 task
invocations, `park` is called with a duration of zero.
This patch adds a new crate: tokio-timer. This crate provides an
efficient timer implemeentation designed for use in Tokio based
applications.
The timer users a hierarchical hashed timer wheel algorithm with six
levels, each having 64 slots. This allows the timer to have a resolution
of 1ms while maintaining O(1) complexity for insert, removal, and firing
of timeouts.
There already exists a tokio-timer crate. This is a complete rewrite
which solves the outstanding problems with the existing tokio-timer
library.
Closes#146.
* Fix races.
This mostly pulls in changes from rust-lang-nursery/futures-rs#881, but
also updates Registration to be a bit more obvious as to what is going
on.
* Reduce spurious wakeups caused by Reactor
This patch adds an ABA guard on token values before registering them
with Mio. This allows catching token reuse and avoid the notification.
This is needed for OS X as the notification is used to determine that a
TCP connect has completed. A spurious notification can potentially cause
write failures.
Currently, the runtime does not shutdown if the runtime handle is
dropped. This can happen during a panic or when the value is simply
dropped.
This patch forces the runtime to shutdown if it is not explicitly
shutdown.
Fixes#209
If a future panics from within the context of a thread pool, the pool
should not be impacted. To do this, polling the future is wrapped with a
catch_unwind. Extra care is taken to ensure that `thread::panicking()`
is set from within the future's drop handle.
Fixes#209
This patch relicenses the Tokio project exclusively under the MIT
license. Before this, the project was dual licensed under MIT and Apache
2. As such, switching to only MIT is permitted.
Fixes#202
This patch fixes a bug where `CurrentThread::turn` is expected to block
even if the executor is idle.
The `turn` API is the low level interface for callers to interact with
the `Sleep` instance used by the `CurrentThread` instance. As such, a
call to `turn` is expected to call `sleep` once if the executor did not
perform any work.
This patch updates `poll_read_ready` to take a `mask` argument, enabling
the caller to specify the desired readiness. `need_read` is renamed to
`clear_read_ready` and also takes a mask.
This enables a caller to listen for HUP events without requiring reading
from the I/O resource.
Currently, if a thread pool instance is dropped without being shutdown,
the workers will run indefinitely. This is not ideal as it leaks the
threadpool.
This patch forces the thread pool to shutdown on drop.
Closes#151
Mio will be requiring `deregister` to be called explicitly in order to
guarantee that Poll releases any state associated with the I/O resource.
See carllerche/mio#753.
This patch adds an explicit `deregister` function to `Registration` and
updates `PollEvented` to call this function on drop.
`Registration::deregister` is also called on `PollEvented::into_inner`.
Closes#168
Some of the benchhmarks were broken and/or using deprecated APIs. This
patch updates the benches and requires them all to compile without
warnings in order to pass CI.
The exampes included in the repository have lagged behind the changes
made. Specifically, they do not use the new runtime construct.
This patch updates examples to use the latest features of Tokio.
Tokio is moving away from using `WouldBlock`, instead favoring
`Async::NotReady`.
This patch updates the TCP and UDP types, deprecating any function that
returns `WouldBlock` and adding a poll_ prefixed equivalent.
Currently, `tokio::spawn` matched the `spawn` function from futures 0.2.
However, this adds additional ergonomic overhead and removes the ability
to spawn from a drop fn. See rust-lang-nursery/futures-rs#830.
This patch switches the behavior to access the thread-local variable
referencing the default executor directly in the `spawn` function.
This allows libraries that require access to reactor related types to
depend on this crate without having to depend on the entirety of Tokio.
For example, libraries that implement their custom I/O resource will
need to access `Registration` or `PollEvented`.
This patch updates the documentation for a number of APIs. It also
introduces a prelude module and an io facade module, re-exporting types
from tokio-io.
Sometimes, passing ownership to an executor is necessary. For example,
some libraries require taking ownership of one.
This patch adds a function that returns an executor associated with a
runtime.
This patch makes a significant change to how I/O resources bind to a
reactor. Currently, an I/O resource (TCP, UDP, PollEvented) will bind
itself with a reactor upon creation.
First, some history.
Originally, tokio-core required that I/O resources be explicitly
associated with a reactor upon creation by passing in a `&Handle`. Tokio
reform introduced a default reactor. If I/O resources do not specify a
reactor upon creation, then the default reactor is used.
However, futures tend to favor being lazy. Creating a future should do
no work, instead it is defining a computation to be performed once the
future is executed. Binding an I/O resource with a reactor on creation
goes against this pattern.
This patch fixes this by allowing I/O resources to lazily bind to a
reactor. An explicit `&Handle` can still be used on creation, but if no
reactor is specified, then the default reactor is used. However, this
binding happens during execution time (read / write) and not creation.
Currently, the thread-local tracking the current thread executor is not
set when a task is dropped. This means that one cannot spawn a new
future from within the drop implementation of another future.
This patch adds support for this by setting the thread-local before
releasing a task.
This implementation is a bit messy. It probably could be cleaned up, but
this is being put off in favor of trying a more comprehensive
reorganization once the current thread executor is feature complete.
The logic that enables `CurrentThread::turn` to avoid unbounded
iteration was incorrect. It was possible for unfortunate timing to
result in a dead lock.
This patch provides a fix as well as a test.
FreeBSD uses a separate kqueue filter type for lio_listio. This change
adds support for that filter type. Full functionality will be provided
by the mio-aio and tokio-file crates.
* Add PollEvented::into_inner
Consumes a PollEvented and returns its inner io object. Useful for io
types that have exclusive ownership of a resource.
See also https://github.com/tokio-rs/tokio-core/commit/9400ffb
CurrentThread::turn uses a turn count strategy to allow `turn` to not
run infinitely. Currently, there is a bug where spawned tasks will not
get executed in calls to `turn`.
This patch fixes the bug by correctly setting the turn count for newly
spawned tasks.
This patch is an intial implementation of the Tokio runtime. The Tokio
runtime provides an out of the box configuration for running I/O heavy
asynchronous applications.
As of now, the Tokio runtime is a combination of a work-stealing thread
pool as well as a background reactor to drive I/O resources.
This patch also includes tokio-executor, a hopefully short lived crate
that is based on the futures 0.2 executor RFC.
* Implement `Park` for `Reactor`
This enables the reactor to be used as the thread parker for executors.
This also adds an `Error` component to `Park`. With this change, a
`Reactor` and a `CurrentThread` can be combined to achieve the
capabilities of tokio-core.
* move src/codec.rs -> src/codec/mod.rs
* Move traits Encoder and Decoder from src/framed_{read|write}.rs into src/codec/{encoder,decoder}.rs
* Move LinesCodec and BytesCodec from src/codecs.rs into src/codec/{lines,bytes}_codec.rs
The `Handle` type is intended to be used by end users of Tokio. The
wakeup functionlity is needed by executor implementations. It doesn't
make sense to put executor specific functionality on a type that is
intended for end users.
* Limit the max number of registered resources
This allows some token space to be used internally. Also, Mio 0.7 will
be limiting the token space some as well.
Mio: https://github.com/carllerche/mio/issues/788Closes#42
Generally speaking, it is unsafe to access to perform asynchronous
operations using `&self`. Taking `&self` allows usage from a `Sync`
context, which has unexpected results.
Taking `&mut self` to perform these operations prevents using these
asynchronous values from across tasks (unless they are wrapped in
`RefCell` or `Mutex`.
The caller should be required to have control of a reactor when setting
it as a fallback. A `Handle` is used to pass into libraries and
functions and should not grant capability of using the associated
reactor as a fallback.
Closes#71
std's `Incoming` iterator yields `TcpStream` instances. This patch
updates the `Incoming` future to match this signature.
This changes the yielded value from `(TcpStream, SocketAddr)` ->
`TcpStream`.
This was added oh-so-long ago and nowadays is just used by one consumer,
`PollEvented`. Let's just inline the already small implementation directly into
`PollEvented` which should make it easier to modify in the future as well
* Refactor UDP SendDgram & RecvDgram
Get rid of unnamed structs in the favor of private structs with named fields
* Change the signature of UdpCodec::encode
Now it is:
```
fn encode(&mut self, msg: Self::Out, buf: &mut Vec<u8>) -> Result<SocketAddr, Self::Error>;
```
Closes https://github.com/tokio-rs/tokio/issues/79
* Fix compilation error from `mio` crate
* Change a return value of reactor::poll to io::Result.
* Revert "Change a return value of reactor::poll to io::Result."
This reverts commit 281d8c32d4.
* Return a result from reactor::poll.
* Drop a reactor if any error occurs. Fix warnings in tests.
* Update a documentation for reactor::turn.
* Unwrap the last turn() call in tests.
This commit removes the `Handle` argument from the following constructors
* `TcpListener::bind`
* `TcpStream::connect`
* `UdpSocket::bind`
The `Handle` argument remains on the various `*_std` constructors as they're
more low-level, but this otherwise is intended to set forth a precedent of by
default not taking `Handle` arguments and instead relying on the global
`Handle::default` return value when necesary.
This method is intended to be used to wake up the reactor from a remote thread
if necessary, forcing it to return from a blocked call of `turn` or otherwise
prevent the next call to `turn` to from blocking.
This commit removes the `Reactor::run` method which has previously been used to
execute futures and turn the reactor at the same time. The tests/examples made
heavy usage of this method but they have now all temporarily moved to `wait()`
until the futures dependency is upgraded. In the meantime this'll allow us to
further trim down the `Reactor` APIs to their final state.
This commit starts to add support for a global event loop by adding a
`Handle::default` method and implementing it. Currently the support is quite
rudimentary and doesn't support features such as shutdown, overriding the return
value of `Handle::default`, etc. Those will come as future commits.
This commit changes the `PollEvented::deregister` signature from
fn deregister(self, handle: &Handle) -> io::Result<()>
to
fn deregister(&self) -> io::Result<()>
Now that the handles are `Send` and `Sync` there's no longer any need to pass it
in (it's already stored in the `PollEvented` itself). Additionally this switches
to `&self` instead of `self` to allow reclamation of the internal resources if
necessary.
This should allow configuration over what reactor accepted streams go on to by
giving back a libstd-bound object that can then be used later in conjunction
with `TcpStream::from_std`.
This commit is targeted at solving tokio-rs/tokio-core#12 and incorporates the
solution from tokio-rs/tokio-core#17. Namely the `need_read` and `need_write`
functions on `PollEvented` now return an error when the connected reactor has
gone away and the task cannot be blocked. This will typically naturally
translate to errors being returned by various connected I/O objects and should
help tear down the world in a clean-ish fashion.
This commit renames the various constructors of networking types to have a
`_std` suffix instead of a smorgasboard of other suffixes, canonicalizing on
`_std` as the suffix for constructors which take the libstd corresponding types.
* small doc cleanups in PollEvented
* small doc cleanups in IoToken
* improve crate level documentation
- Add links to the futures, mio and tokio-uds crates.
- Add links to various structs and types mentioned.
- use eprintln for error reporting in the example.
* improvements to the UdpSocket documentation
- Fixed links usage.
- Removed references to a no longer existing `Window` struct.
- Made notes about using functions in context of a future.
* documentation improvements to UdpFramed and UdpCodec
- Since HTTP uses TCP (QUIC aside) using it as an example in an UDP
protocol feels wrong.
- Make the note of tampering with the underlying streams more explicit.
* update reactor module level documentation
Adds an explanation of every public struct.
* expand Handle and Remote documentation
* expand net module documentation
Adds an explanation of every public struct and how they work together.
* update TcpListener documentation
Reorder the various option methods; get first then set.
Note about panicing added to poll_read.
* remove mention of none-existing future R
* improve documentation of TcpStream
* fix UdpSocket doc
This when wrong when merging various commits.
* Make Handle `Send + Sync`.
This is an initial implementation making `Handle: Send + Sync`. It uses
a `RwLock` to coordinate access to the underlying state storage. An
implementation without the lock is left to later.
This pass also leaves a lot of dead code that can be removed in later
commits.
* Remove reactor code related to message passing
The previous commit removed the need for using message passing to
communicate with the reactor. This commit removes all the unnecessary
code.
Mio should be providing the compatibility layer so there is no need to
run CI on OS X as well as Linux.
Benches are currently being skipped as the Tokio reform work progresses.
In accordance with tokio-rs/tokio-rfcs#3, the executor functionality of
Tokio is being removed and will be relocated into futures-rs as a
"current thread" executor.
This PR removes task execution from the code base. As a temporary
mesure, all examples and tests are switched to using CpuPool.
Depends on #19.
In accordance with tokio-rs/tokio-rfcs#3, timers are being extracted
from Tokio and moved to a separate crate (probably futures-timer).
This PR removes timers from the code base.
This text historically was copied verbatim from rust-lang/rust's own README [1]
with the intention of licensing projects the same as rustc's own license, namely
a dual MIT/Apache-2.0 license. The clause about "various BSD-like licenses"
isn't actually correct for almost all projects other than rust-lang/rust and
the wording around "both" was slightly ambiguous.
This commit updates the wording to match more precisely what's in the
standard library [2], namely clarifying that there aren't any BSD-like licenses
in this repository and that the source is licensable under either license, at
your own discretion.
[1]: https://github.com/rust-lang/rust/tree/f0fe716dbcbf2363ab8f929325d32a17e51039d0#license
[2]: https://github.com/rust-lang/rust/blob/f0fe716dbcbf2363ab8f929325d32a17e51039d0/src/libstd/lib.rs#L5-L9
This example is intended to showcase sharing state between all connected clients
on a server, for example a key/value store (in-memory database)
Closes#257
Some contexts, like EC2, have `Instant::now` as a relatively expensive function
to call. To help amortize the cost of this function this commit exposes a new
function on `Interval` and `Timeout` to pass in the assumed current time.
The `consume_queue` function can be relatively slow for an empty queue (the fast
path) so optimize this a bit with a preflight check that should just touch a few
atomics.
My slab bump in #246 added a potential new panic in Core's Inner's
`cancel_timeout`.
Thankfully, the only way `cancel_timeout` can be called is from
TimeoutToken's `cancel_timeout`, which is crate-internal only and is
only called from Timeout's and Interval's drop fn's.
This change simply adds new clarifying documentation around
TokenTimeout's cancel_timeout to "future proof" anybody looking to use
cancel_timeout directly (not just on drop).
This patch disables various Unix-specific platform features that are
not enabled on Fuchsia. It also updates the mio version to 0.6.10,
which is the first release that supports Fuchsia.
This commit adds a general-purpose method for querying the readiness of a
`PollEvented` type. This new method, `poll_ready`, takes a blanket `mio::Ready`
and tests if any part of it is ready. The purpose of this is to expose
platform-specific events through `PollReady` such as `hup` and `error` events
other than just the platform-agnostic readable/writable events.
The semanatics of this method are:
* The `poll_ready` function takes a mask, and the return value is either
`Async::Ready` with a subset of these events that are ready or `None` if none
of them are ready.
* There can be up to two tasks blocked on a `PollEvented`, so we need to pick
which one is suitable for these new events. Currently all events are routed to
the `read` task unless the writable bit is set. This is mostly only relevant
for multi-task usage or if you're manually calling `need_read` and/or
`need_write`, and hopefully the docs will cover this now.
If the underlying read_bufs() call returned a WouldBlock error,
TcpStream.read_buf() was erroneously calling self.io.need_write(), when
it should actually call self.io.need_read().
* if remaining bytes are smaller then 8 * 1024, allocate 8 * 1024
* otherwise allocate as much as needed to hold the remaining bytes
without re-allocations.
The previous implementation would always use the capacity
of the previous buffer, which would effectively prevent it
from ever shrinking.
This also means that protocol with greater variance in
possible frame sizes would likely be heavily over-allocating.
If these implementations use zero-copy, this would imply
that even small frames kept alive by the client would use
large amounts of memory.
The change is motivated by the implementation of the
cassandra-protocol, which allows frames of up to 256MB
in size, which solely depend on the kind of query.
Using Linux's `epoll(7)` interface, a `EPOLLHUP` condition is
signalled for the reading end of a pipe or socket when the other end
is closed for writing. This may happen in combination with `EPOLLIN`
being signalled (if further data is available for reading), or with
`EPOLLIN`.
If `EPOLLHUP` is signalled without `EPOLLIN` it indicates an immediate
EOF condition, which will result in the next `read()` suceeding with 0
bytes read. It is thus not required to handle `EPOLLHUP` specially in
the reader, but we need to indicate readiness upon encountering it.
Looking at the `kqueue` and `windows` mio backends, they seem to be
turn a detected EOF and connection reset into `mio::Ready::hup()`
events, so it may be reasonable to speculate that this change is an
improvement for these platforms as well.
Fixes#117.
Note that this brings in a lot of machinery just for the sake of
wrapping file descriptors as PollEvented. This should maybe be moved
into the public API of some crate.
* Move to `std::net` as it's all purely UDP related
* Rename to `UdpCodec` and `UdpFramed` to give a consistent `Udp` prefix
* Add `RecvDgram`, rename `SendDGram` to `SendDgram`
* Touch up some style here and there
This commit makes a few changes to the Sink implementation on Framed:
* Backpressure is implemented for `start_send`. If the write buffer is
over 8KiB and can't be flushed, no new items are accepted.
* 0 length writes to the upstream transport are translated into a
`WriteZero` error, as with `io::Write::write_all`. `write_all` checks
for and ignores `Interrupted` errors, but I do not think this is
necessary for non-blocking writes.
* In `poll_complete`, the upstream transport is not flushed until
*after* writing the entire write buffer.
traits into `Codec`
A previous commit refactored such that `Encode` and `Decode` are
implemented directly on the types being encoded or decoded. This was
thought to be less expressive but more convenient than having a separate
notion of a (stateful) encoder or decoder.
However, there are certain situations where the approach is just too
limiting: you're required to implemented `Decode` and `Encode` for types
you don't "own" and can't newtype.
This commit moves back to a setup where `Self` represents the
encoder/decoder state; it also merges the two traits into a single
`Codec` trait, since they are currently always used together.
- Gets rid of `easy` module, instead providing framing support directly
in the `io` module.
- In particular, adds a framing adapter directly to the `Io` trait,
which gives you a Stream + Sink object. That object can then be
`split` into separate `Stream` and `Sink` objects if needed.
- Deprecates the `FramedIo` trait; that's now just Stream + Sink.
- Updates the line framing test to use the stream/sink combinators.
Typically this happens automatically as the `E` in `PollEvented<E>` is an owned
reference (e.g. a `TcpStream`) where dropping that will close the resource,
automatically unregistering it from the event loop. In some situations, however,
this isn't always the case, so the deregistering needs to happen manually.
This commit makes a few tweaks to the new `easy` module:
- Rename `Parse` to `Decode`, and `Serialize` to `Encode`.
- Don't use `Poll` for the `decode` method; we prefer to reserve
that type for actual aync events, and in particular for a `NotReady`
result to imply that some task scheduling has taken place. Instead,
use an internal `Option`.
This commit extracts the concrete implementation of `FrameIo` in tokio-proto to
tokio-core under the name `EasyFramed`. This extraction is accompanied with a
new `EasyBuf` buffer type to work with when parsing types.
The purpose of this movement is to provide a clear and easy entry point at the
`FramedIo` layer for those who need it. Eventually these buffer types will get
replaced or moved to the `bytes` crate, but in the interest of an 0.1 release
and remaining backwards compatible with the tokio-core 0.1 release this is
adding a separate module.
The other read futures (read_exact, read_until, etc.) all expose their
concrete future types so that function signatures can return them, but
until now `read()` didn't. Exposing it with the name "Read" causes
naming conflicts with the std::io::Read trait, so the easiest thing to
do is to just change the name. Importing std::io::Read under a different
name would've been an option too, but that would probably be annoying
for consumers in the same way it's annoying for us.
The original PR (https://github.com/tokio-rs/tokio-core/pull/29) decided
that "read" was a better name than "read_some", so I'm leaving the top
level functions unchanged. I don't have a strong opinion about it one
way or the other, but I *do* think it's worth bikeshedding a little bit.
Python's asyncio library actually ended up with a very similar issue
around naming inconsistency between the sync and async worlds, and we
can hopefully avoid repeating that: https://bugs.python.org/issue22279
* Send source address of message in addition to the message to connected clients.
* Move `spawn_fn` to the bottom.
* Use `map` instead of `and_then` if there is no need for blocking.
* `map` to unit where values are not needed anymore.
In general it's easier to implement and should have more predictable performance
semantics for applications in general. More serious timer usage can go through
`tokio-timer` which has properly configurable timer wheels and such.
Closes#2Closes#7
* Handle -> Remote
* Pinned -> Handle
All APIs now take a `&Handle` by default and in general can return an immediate
`io::Result` instead of an `IoFuture`. This reflects how most usage will likely
be done through handles rather than remotes, and also all previous functionality
can be recovered with a `oneshot` plus `Remote::spawn`.
Closes#15
* Remove `LoopData` as it's no longer necessary
* Add `LoopHandle::spawn` to spawn new futures onto an event loop
* Add `LoopData::spawn` to also spawn new futures onto an event loop
* Rejigger the implementation of the event loop a bit (make a slab of futures),
but otherwise everything else is pretty constant.
Split it up into a number of targeted modules for each purpose, for example loop
data, I/O sources, timeouts, and channels. No actual change is intended to be
part of this commit.
This commit contains a few refactorings, but the major goal is to remove the
`Arc` that's stored inside of each `ReadinessStream` and `Scheduled` slot in the
event loop. The original purpose of this `Arc` was to share the I/O object among
the concrete handle itself and the event loop. The event loop would then change
how the socket is registered over time and then deregister it when it gets a
"shutdown request".
Nowadays, however, once an I/O object is registered with the event loop it's
never updated. Additionally, we don't actually need to call `deregister` but can
rather just instead close the I/O object itself and let the kernel/event loop
take care of the cleanup. All we need to do on deregistering is free up the slab
entry.
The major result of this commit is that I/O objects no longer need to be `Sync`
(as they're not stored in an `Arc`). Instead they just need to be `Send +
'static` as one might otherwise expect.
Along the way this also refactors a few pieces here and there to make more sense
in this new scheme. The `ReadinessStream` type now has a type parameter
indicating an owned reference to the I/O object it wraps. This can be accessed
via the `get_ref` and `get_mut` methods. Additionally I/O tokens on the event
loop are now a full-fledged `IoToken` type which we can change in the future if
we need to.
We know that the future will never persist beyond this stack frame, so we can
just leave its ownership on the stack frame itself and receive notifications off
the event loop that we need to poll it.
The core trait and associated types no longer require the `'static` bounds, and
all associated methods have also had the `'static` bounds removed. The only
location for the `'static` bound is `forget`.
While the restriction of `'static` on `forget` is here to stay, we'll soon
enable `Loop::run` to take a non-`'static` future, allowing driving a
non-`'static` future.
Also involved yet another round of bug fixes to the timer wheel as well as an
unfortunately serious rejiggering of the level-translation into libcurl. Eew.
* Auto-register interest whenever we see WouldBlock
* Remove implementations of `Stream<Item=Ready>`, no longer needed
* Add explicit `poll_{read,write}` methods, if needed
* Remove all I/O streams, libstd ones suffice
* Update all I/O futures
A more appealing model is actually just automatically inferring what needs to be
scheduled based on what actions are done during poll. For example if during a
poll you check a oneshot channel, then the current task is registered for being
woken up if it's not ready. Similarly this will apply to I/O where if I/O is
attempted but we see EAGAIN then we'll schedule the task to get notified when
it's ready.
This may also have performance benefits in some niche situations because you
don't need to recompute where you are in the state machine both during poll and
during schedule. Instead, it now happens all at once.
This bound existed for two primary reasons, both detail below, and both of which
have now been solved.
One of the primary reasons this existed was due to the presence of `tailcall`.
Each standard combinator will call `tailcall` as appropriate, storing the
resulting trait object. Storing trait objects influences the applicatoin of the
`Send` and `Sync` bounds normally, but a key insight here is that we're not
storing trait objects but rather just pieces of otherwise internal futures.
With this insight the main storage for these futures, `Collapsed`, could simply
implement `Send` so long as the future itself originally implemented `Send`.
This in turn means that `tailcall` must be an `unsafe` method, but it seems well
worth the benefit of relaxing the `Send` bound.
The second primary reason for this bound was so the `Task` itself could be send.
This is critical for ensuring that futures can receive notifications from
multiple threads (e.g. be a future waiting on sources of multiple events).
Another key insight here, however, is that only the *outer* future needs to be
`Send`. We already have a solution, with `LoopData`, to make non-`Send` data
`Send`. By implementing `Future` directly for `LoopData<F: Future>`, this means
that it's trivial to make any future sendable by simply pinning it to an event
loop!
With these two pieces combined, it means that `Send` is no longer needed as a
bound on the `Future` and `Stream` traits. It may practically mean that
`LoopData` is used commonly in some scenarios, but that's quite a small price to
pay for relaxing the requirements of the core trait.
Some other ramifications of this change are:
* The `Future::boxed` and `Stream::boxed` methods now require `Self` to adhere
to `Send`. This is expected to be the most common case, and in the less common
case of not-`Send` `Box::new` can be used.
* Two new type aliases, `BoxFuture` and `BoxStream` have been introduced to
assist in writing APIs that return a trait object which is `Send`. Both of
these type aliases package in the `Send` bound.
* A new `LoopPin` type, added in the previous commit, can be used to easily
generate handles that can be used to pin futures to an event loop without
having a literal reference to the event loop itself.
It's already Sync because it's safe to access amongst many threads by the sheer
fact that data is only accessed on one thread. That is, all other concurrent
accessors, if any, will receive `None`.
This type acts for a handle to storage of non-`Send` data. The handle itself is
sendable across threads and is therefore suitable for storage in a `Future`.
This data uses communication internally and a new method on `Task` to ensure
that when the data needs to be accessed the future will find its way to the
right thread.
More on this type coming soon!
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Alice Ryhl