Compare commits

...
Author SHA1 Message Date
Carl Lerche 98c9a77f18 prepare v0.2.3 release (#1912) 2019-12-06 09:47:28 -08:00
Carl Lerche e00c49611a doc: fix TcpListener example to compile (#1911)
The `process_socket` is hidden from the user which makes the example
fail to compile if copied by the reader.
2019-12-06 09:16:08 -08:00
Jeremy Kolb 9c9fabc44b Close markdown (#1910) 2019-12-06 07:51:24 -08:00
Steven Fackler c3461b3ef3 time: impl From between std / tokio Instants (#1904) 2019-12-05 12:03:04 -08:00
Eliza Weisman b7ecd35036 task: fix LocalSet failing to poll all local futures (#1905)
Currently, a `LocalSet` does not notify the `LocalFuture` again at the
end of a tick. This means that if we didn't poll every task in the run
queue during that tick (e.g. there are more than 61 tasks enqueued),
those tasks will not be polled.

This commit fixes this issue by changing `local::Scheduler::tick` to
return whether or not the local future needs to be notified again, and
waking the task if so.

Fixes #1899
Fixes #1900

Signed-off-by: Eliza Weisman <[email protected]>
2019-12-05 12:00:10 -08:00
Kevin Leimkuhler dbcd1f9a09 time: Remove HandlePriv (#1896)
## Motivation

#1800 removed the lazy binding of `Delay`s to timers. With the removal of the
logic required for that, `HandlePriv` is no longer needed. This PR removes the
use of `HandlePriv`.

A `TODO` was also removed that would panic if when registering a new `Delay`
the current timer handle was full. That has been fixed to now immediately
transition that `Delay` to an error state that can be handled in a similar way
to other error states.

Signed-off-by: Kevin Leimkuhler <[email protected]>
2019-12-04 20:46:16 -08:00
Eliza Weisman 0e729aa341 task: fix infinite loop when dropping a LocalSet (#1892)
## Motivation

There's currently an issue in `task::LocalSet` where dropping the local
set can result in an infinite loop if a task running in the local set is
notified from outside the local set (e.g. by a timer). This was reported
in issue #1885.

This issue exists because the `Drop` impl for `task::local::Scheduler`
does not drain the queue of tasks notified externally, the way the basic
scheduler does. Instead, only the local queue is drained, leaving some
tasks in place. Since these tasks are never removed, the loop that
continues trying to cancel tasks until the owned task list is totally
empty continues infinitely.

I think this issue was due to the `Drop` impl being written before a
remote queue was added to the local scheduler, and the need to close the
remote queue as well was overlooked.

## Solution

This branch solves the problem by clearing the local scheduler's remote
queue as well as the local one.

I've added a test that reproduces the behavior. The test fails on master
and passes after this change.

In addition, this branch factors out the common task queue logic in the
basic scheduler runtime and the `LocalSet` struct in `tokio::task`. This
is because as more work was done on the `LocalSet`, it has gotten closer
and closer to the basic scheduler in behavior, and factoring out the
shared code reduces the risk of errors caused by `LocalSet` not doing
something that the basic scheduler does. The queues are now encapsulated
by a `MpscQueues` struct in `tokio::task::queue` (crate-public).  As a
follow-up, I'd also like to look into changing this type to use the same
remote queue type as the threadpool (a linked list).

In particular, I noticed the basic scheduler has a flag that indicates
the remote queue has been closed, which is set when dropping the
scheduler. This prevents tasks from being added after the scheduler has
started shutting down, stopping a potential task leak. Rather than
duplicating this code in `LocalSet`, I thought it was probably better to
factor it out into a shared type.

There are a few cases where there are small differences in behavior,
though, so there is still a need for separate types implemented _using_
the new `MpscQueues` struct. However, it should cover most of the 
identical code.

Note that this diff is rather large, due to the refactoring. However, the
actual fix for the infinite loop is very simple. It can be reviewed on its own
by looking at commit 4f46ac6. The refactor is in a separate commit, with
the SHA 90b5b1f.

Fixes #1885

Signed-off-by: Eliza Weisman <[email protected]>
2019-12-04 11:20:57 -08:00
Artem Vorotnikov cbe369a3ed Make JoinError Sync (#1888)
* Make JoinError Sync

* Move Mutex inside JoinError internals, hide its constructors

* Deprecate JoinError constructors, fix internal usages
2019-12-04 10:51:23 -08:00
Juan Alvarez 8bcbe78dbe remove io workarounds from example (#1891)
This PR removes no longer needed io workarounds from connect example.
2019-12-03 16:07:09 -08:00
Christopher Coverdale 6efe07c3fb Fixing minor spelling mistake in task docs (#1889) 2019-12-03 12:01:59 -08:00
Xinkai Chen 8a2160a913 Add unit tests for tokio::File::AsRaw{Fd,Handle} for Unix and Windows. (#1890)
Supersedes #1640.
2019-12-03 09:56:32 -08:00
baizhenxuan 38c361781f examples: fix tinyhttp (#1884) 2019-12-02 20:28:36 -08:00
Eliza Weisman 07451f8b94 task: relax 'static bound in LocalSet::block_on (#1882)
## Motivation

Currently, `tokio::task::LocalSet`'s `block_on` method requires the
future to live for the 'static lifetime. However, this bound is not
required — the future is wrapped in a `LocalFuture`, and then passed
into `Runtime::block_on`, which does _not_ require a `'static` future.

This came up while updating `tokio-compat` to work with version 0.2. To
mimic the behavior of `tokio` 0.1's `current_thread::Runtime::run`, we
want to be able to have a runtime block on the `recv` future from an
mpsc channel indicating when the runtime is idle. To support `!Send`
futures, as the old `current_thread::Runtime` did, we must do so inside
of a `LocalSet`. However, with the current bounds, we cannot await an
`mpsc::Receiver`'s `recv` future inside the `LocalSet::block_on` call.

## Solution

This branch removes the unnecessary `'static` bound.

Signed-off-by: Eliza Weisman <[email protected]>
2019-12-02 16:43:33 -08:00
Carl Lerche e87df0557d io: add async fns for reading / writing bufs (#1881)
Adds `read_buf` and `write_buf` which work with `T: BufMut` and `T: Buf`
respectively. This adds an easy API for using the buffer traits provided
by `bytes.
2019-12-02 13:09:31 -08:00
Carl Lerche a8a4a9f0fc blocking: fix spawn_blocking after shutdown (#1875)
The task handle needs to be shutdown explicitly and not dropped.

Closes #1853
2019-12-01 12:58:01 -08:00
Carl Lerche 8b60c5386a doc: fix documented feature flags for tokio::task (#1876)
Some feature flags are missing and some are duplicated.

Closes #1836
2019-12-01 12:49:38 -08:00
Carl Lerche af07f5bee7 sync: expand oneshot docs and TryRecvError (#1874)
`oneshot::Receiver::try_recv` does not provide any information as to the
reason **why** receiving failed. The two cases are that the channel is
empty or that the channel closed.

`TryRecvError` is changed to be an enum of those two cases. This is
backwards compatible as `TryRecvError` was an opaque struct.

This also expands on `oneshot` API documentation, adding details and
examples.

Closes #1872
2019-12-01 10:48:47 -08:00
Ivan Petkov 939a0dd7b0 process: rewrite and simplify the issue_42 test (#1871) 2019-11-30 15:17:04 -08:00
Carl Lerche 1ea6733568 io: read/write big-endian numbers (#1863)
Provide convenience methods for encoding and decoding big-endian numbers
on top of asynchronous I/O streams. Only primitive types are provided
(24 and 48 bit numbers are omitted).

In general, using these methods won't be the fastest way to do
encoding/decoding with asynchronous byte streams, but they help to get
simple things working fast.
2019-11-30 13:13:21 -08:00
Carl Lerche 8ce408492a doc: improve AsyncBufReadExt API documentation (#1868)
Remove "old" docs that were left over during a rewrite, add examples and
additional details.
2019-11-30 13:12:39 -08:00
Carl Lerche b559a0cd9a net: expose TcpStream::poll_peek (#1864)
This used to be exposed in 0.1, but was switched to private during the
upgrade. The `async fn` is sufficient for many, but not all cases.

Closes #1556
2019-11-30 09:36:03 -08:00
Carl Lerche 417460cf86 doc: expand mpsc::Sender::send API documentation (#1865)
Includes more description, lists errors, and examples.

Closes #1579
2019-11-30 09:35:23 -08:00
Carl Lerche adaba1a0bc doc: add API docs for AsyncBufReadExt::read_line (#1866)
Include more details and an example.

Closes #1592
2019-11-30 09:34:42 -08:00
Ivan Petkov 467b6ea783 chore: prepare v0.2.2 release (#1857) 2019-11-29 11:09:28 -08:00
Carl Lerche a2cfc877a7 rt: fix basic_scheduler notification bug (#1861)
The "global executor" thread-local is to track where to spawn new tasks,
**not** which scheduler is active on the current thread. This fixes a
bug with scheduling tasks on the basic_scheduler by tracking the
currently active basic_scheduler with a dedicated thread-local variable.

Fixes: #1851
2019-11-29 10:23:22 -08:00
Ömer Sinan Ağacan ec7f2ae306 docs: Mention features for basic_scheduler, threaded_scheduler (#1858)
Fixes #1829
2019-11-29 08:26:58 -08:00
Bartek Iwańczuk 4261ab6627 fs: add File::into_std and File::try_into_std methods (#1856)
In version 0.1 there was File::into_std method that destructured
tokio_fs::File into std::fs:File. That method was lacking in
version 0.2.

Fixes: #1852
2019-11-28 17:09:28 -08:00
Ivan Petkov aef434c089 signal: update documentation with caveats (#1854) 2019-11-28 15:03:04 -08:00
Ömer Sinan Ağacan cd73951130 Implement Stream for signal::unix::Signal (#1849)
Refs #1848
2019-11-28 08:54:50 -08:00
Eliza Weisman 524e66314f task: fix panic when dropping LocalSet (#1843)
It turns out that the `Scheduler::release` method on `LocalSet`'s
`Scheduler` *is* called, when the  `Scheduler` is dropped with tasks
still running. Currently, that method is `unreachable!`, which means
that dropping a `LocalSet` with tasks running will panic.

This commit fixes the panic, by pushing released tasks to
`pending_drop`. This is the same as `BasicScheduler`.

Fixes #1842
2019-11-27 14:24:44 -08:00
Michael Zeller 34d751bf92 net: fix ucred for illumos/solaris (#1772) 2019-11-27 12:22:22 -08:00
Oleg Nosov 942feab040 doc: misc API documentation fixes (#1834) 2019-11-27 12:05:42 -08:00
Oleg Nosov dc356a4158 doc: fix runtime::Builder example (#1841) 2019-11-27 12:03:57 -08:00
Oleg Nosov 2cd1d74092 rt: specify that runtime should have task scheduler (#1839)
* Specify that runtime should have task scheduler

* Even more detailed panic message for incorrect task spawn
2019-11-27 10:25:21 -08:00
Carl Lerche 632ee507ba prepare v0.2.1 release (#1832)
This includes `task::LocalSet` as well as some misc small fixes.
2019-11-26 21:46:02 -08:00
Carl Lerche 7f605ee27f doc: fix and improve incoming() API doc (#1831)
This fixes the API docs for both `TcpListener::incoming` and
`UnixListener::incoming`. The function now takes `&mut self` instead of
`self`. Adds an example for both function.
2019-11-26 21:15:13 -08:00
Eliza Weisman 38e602f4d8 task: add LocalSet API for running !Send futures (#1733)
## Motivation

In earlier versions of `tokio`, the `current_thread::Runtime` type could
be used to run `!Send` futures. However, PR #1716 merged the
current-thread and threadpool runtimes into a single type, which can no
longer run `!Send` futures. There is still a need in some cases to
support futures that don't implement `Send`, and the `tokio-compat`
crate requires this in order to provide APIs that existed in `tokio`
0.1.

## Solution

This branch implements the API described by @carllerche in
https://github.com/tokio-rs/tokio/pull/1716#issuecomment-549496309. It
adds a new `LocalSet` type and `spawn_local` function to `tokio::task`.
The `LocalSet` type is used to group together a set of tasks which must
run on the same thread and don't implement `Send`. These are available
when a new "rt-util" feature flag is enabled.

Currently, the local task set is run by passing it a reference to a
`Runtime` and a future to `block_on`. In the future, we may also want
to investigate allowing spawned futures to construct their own local
task sets, which would be executed on the worker that the future is
executing on. 

In order to implement the new API, I've made some internal changes to
the `task` module and `Schedule` trait to support scheduling both `Send`
and `!Send` futures.

Signed-off-by: Eliza Weisman <[email protected]>
2019-11-26 17:03:18 -08:00
Artem Vorotnikov 8e83a9f2c3 chore: replace Gitter badge with Discord (#1828) 2019-11-26 16:00:38 -08:00
Carl Lerche c146f48f0b fs: impl AsRawFd / AsRawHandle for File (#1827)
This provides the ability to get the raw OS handle for a `File`. The
`Into*` variant cannot be provided as `File` needs to maintain ownership
of the `File`. The actual handle may have been moved to a background
thread.
2019-11-26 16:00:26 -08:00
Benjamin Fry ebf5f37989 time: reexport Elapsed (#1826) 2019-11-26 15:10:41 -08:00
Carl Lerche abfa857f09 chore: remove updating note from readme (#1824) 2019-11-26 10:36:17 -08:00
Carl Lerche a81e2722a4 chore: prepare v0.2.0 release (#1822) 2019-11-26 09:17:27 -08:00
Carl Lerche 4ddc437170 doc: add more doc_cfg annotations (#1821)
Also makes the `tokio::net::{tcp, udp, unix}` modules only for "utility"
types. The primary types are in `tokio::net` directly.
2019-11-25 14:32:55 -08:00
Carl Lerche 3ecaa6d91c docs: improve tokio::io API documentation (#1815)
Adds method level documentation for `tokio::io`.
2019-11-23 08:24:03 -08:00
leo-lb 0bc68adb34 tokio: remove performance regression notice (#1817) 2019-11-23 07:45:56 -08:00
Ivan Petkov e20dff39ce process: do not kill spawned processes on drop (#1814)
This updates the tokio `Command` and `Child` behavior to match that of
the stdlib: spawned processes will *not* be automatically killed when
the handle is dropped

Unlike the stdlib, any dropped (unix) processes may be reaped by tokio
behind-the-scenes after they exit and if new processes are awaited,
which mitigates the risks of piling up unreaped zombie unix processes

A `Command::kill_on_drop` method is added to allow the caller to
control whether the spawned child should be killed when the handle is
dropped. By default, this value is `false`.

The `Child::forget` method has been removed, as it is superseded by
`Command::kill_on_drop`
2019-11-22 20:10:05 -08:00
Carl Lerche 7b4c999341 default all feature flags to off (#1811)
Changes the set of `default` feature flags to `[]`. By default, only
core traits are included without specifying feature flags. This makes it
easier for users to pick the components they need.

For convenience, a `full` feature flag is included that includes all
components.

Tests are configured to require the `full` feature. Testing individual
feature flags will need to be moved to a separate crate.

Closes #1791
2019-11-22 15:55:10 -08:00
Carl Lerche e1b1e216c5 ci: bring back build tests (#1813)
This directory was deleted when `cargo hack` was introduced, however
there were some tests that were still useful (macro failure output).

Also, additional build tests will be added over time.
2019-11-22 14:38:49 -08:00
Taiki Endo 7cd63fb946 ci: use -Z avoid-dev-deps in features check instead of --no-dev-deps (#1812) 2019-11-22 14:13:18 -08:00
Carl Lerche bf741fec35 ci: generate docs (#1810)
Check docs as part of CI. This should catch link errors.
2019-11-22 11:55:57 -08:00
Carl Lerche 9b2aa14bb1 docs: annotate io mod with doc_cfg (#1808)
Annotates types in `tokio::io` module with their required feature flag.
This annotation is included in generated documentation.

Notes:

* The annotation must be on the type or function itself. Annotating just
  the re-export is not sufficient.

* The annotation must be **inside** the `pin_project!` macro or it is
  lost.
2019-11-22 09:56:08 -08:00
Carl Lerche 8546ff826d runtime: cleanup and add config options (#1807)
* runtime: cleanup and add config options

This patch finishes the cleanup as part of the transition to Tokio 0.2.
A number of changes were made to take advantage of having all Tokio
types in a single crate. Also, fixes using Tokio types from
`spawn_blocking`.

* Many threads, one resource driver

Previously, in the threaded scheduler, a resource driver (mio::Poll /
timer combo) was created per thread. This was more or less fine, except
it required balancing across the available drivers. When using a
resource driver from **outside** of the thread pool, balancing is
tricky. The change was original done to avoid having a dedicated driver
thread.

Now, instead of creating many resource drivers, a single resource driver
is used. Each scheduler thread will attempt to "lock" the resource
driver before parking on it. If the resource driver is already locked,
the thread uses a condition variable to park. Contention should remain
low as, under load, the scheduler avoids using the drivers.

* Add configuration options to enable I/O / time

New configuration options are added to `runtime::Builder` to allow
enabling I/O and time drivers on a runtime instance basis. This is
useful when wanting to create lightweight runtime instances to execute
compute only tasks.

* Bug fixes

The condition variable parker is updated to the same algorithm used in
`std`. This is motivated by some potential deadlock cases discovered by
`loom`.

The basic scheduler is fixed to fairly schedule tasks. `push_front` was
accidentally used instead of `push_back`.

I/O, time, and spawning now work from within `spawn_blocking` closures.

* Misc cleanup

The threaded scheduler is no longer generic over `P :Park`. Instead, it
is hard coded to a specific parker. Tests, including loom tests, are
updated to use `Runtime` directly. This provides greater coverage.

The `blocking` module is moved back into `runtime` as all usage is
within `runtime` itself.
2019-11-21 23:28:39 -08:00
Eliza Weisman 6866fe426c docs: expand and update crate-level docs (#1806)
## Motivation

Tokio's crate-level docs are currently pretty sparse, and in some cases
reference old names for APIs. Before 0.2 is released, they could use a
fresh coat of paint.

## Solution

This branch reworks and expands the `lib.rs` docs. In particular, I've
added a new "A Tour of Tokio" section, inspired by the [standard
library's similarly-named section][std]. This section lists all of
`tokio`'s public modules, and summarizes their major APIs. It also lists
the feature flags necessary to enable those APIs.

[std]: https://doc.rust-lang.org/std/index.html#a-tour-of-the-rust-standard-library

Signed-off-by: Eliza Weisman <[email protected]>
2019-11-21 14:09:10 -08:00
Eliza Weisman d88846c4eb docs: update and expand the tokio::runtime API docs (#1804)
## Motivation

The `tokio::runtime` module's docs need to be updated to
track recent changes.

## Solution

This branch updates and expands the `runtime` docs.

Signed-off-by: Eliza Weisman <[email protected]>
2019-11-20 17:46:35 -08:00
Eliza Weisman 7e6a10fccd docs: refresh tokio::io API docs (#1803)
## Motivation

The `tokio::io` module's docs are fairly sparse and not particularly up
to date. They ought to be improved before release.

## Solution

This branch adds new module-level docs to `tokio::io`. The new docs are
largely inspired by `std::io`'s documentation, and highlight the
similarities and differences between `tokio::io` and `std::io`.

Signed-off-by: Eliza Weisman <[email protected]>
2019-11-20 15:09:38 -08:00
Carl Lerche 502cf5d95c io: flatten split module (#1802) 2019-11-20 14:45:38 -08:00
Eliza Weisman c223db3589 docs: improve tokio::task API documentation (#1801)
## Motivation

The new `tokio::task` module is pretty lacking in API docs. 

## Solution

This branch adds new API docs to the `task` module, including:

* Module-level docs with a summary of the differences between 
  tasks and threads
* Examples of how to use the `task` APIs in the module-level docs
* More docs for `yield_now`
* More docs and examples for `JoinHandle`, based on the 
  `std::thread::JoinHandle` API docs.

This branch contains commits cherry-picked from #1794 

Signed-off-by: Eliza Weisman <[email protected]>
2019-11-20 14:36:45 -08:00
Carl Lerche 5cd665afd7 chore: update bytes dependency to git master (#1796)
Tokio will track changes to bytes until 0.5 is released.
2019-11-20 14:27:49 -08:00
Kevin Leimkuhler 3e643c7b81 time: Eagerly bind delays to timer (#1800)
## Motivation

Similar to #1666, it is no longer necessary to lazily register delays with the
executions default timer. All delays are expected to be created from within a
runtime, and should panic if not done so.

## Solution

`tokio::time` now assumes there to be a `CURRENT_TIMER` set when creating a
delay; this can be assumed if called within a tokio runtime. If there is no
current timer, the application will panic with a "no current timer" message.

## Follow-up

Similar to #1666, `HandlePriv` can probably be removed, but this mainly prepares
for 0.2 API changes. Because it is not in the public API, this can be done in a
following change.

Signed-off-by: Kevin Leimkuhler <[email protected]>
2019-11-20 12:24:41 -08:00
Pen Tree bc150cd0b5 Fix doc links (#1799)
Link fix only. After this fix, `cargo doc --package` succeeds.
2019-11-20 12:24:17 -08:00
Carl Lerche 15dce2d11a net: flatten split mod (#1797)
The misc `split` types (`ReadHalf`, `WriteHalf`, `SendHalf`, `RecvHalf`)
are moved up a module and the `*::split` module is removed.
2019-11-20 11:29:32 -08:00
Taiki Endo d4fec2c5d6 chore: enable feature flag check on windows (#1798) 2019-11-20 07:05:50 -08:00
Carl Lerche 69975fb960 Refactor the I/O driver, extracting slab to tokio::util. (#1792)
The I/O driver is made private and moved to `tokio::io::driver`. `Registration` is
moved to `tokio::io::Registration` and `PollEvented` is moved to `tokio::io::PollEvented`.

Additionally, the concurrent slab used by the I/O driver is cleaned up and extracted to
`tokio::util::slab`, allowing it to eventually be used by other types.
2019-11-20 00:05:14 -08:00
Carl Lerche 7c8b8877d4 runtime: fix lost wakeup bug in scheduler (#1788)
When checking if a worker needs to be unparked, the SeqCst load does not
provide the necessary synchronization to ensure the scheduled task is
visible to the searching worker. The `load` is switched to
`fetch_add(0)` which does establish the necessary synchronization.

Adding unit tests catching this bug will require a fix to loom and will
be done at a later time. The bug fix has been validated with manual
testing.

Fixes #1768
2019-11-19 08:01:46 -08:00
Carl Lerche 0d38936b35 chore: refine feature flags (#1785)
Removes dependencies between Tokio feature flags. For example, `process`
should not depend on `sync` simply because it uses the `mpsc` channel.
Instead, feature flags represent **public** APIs that become available
with the feature enabled. When the feature is not enabled, the
functionality is removed. If another Tokio component requires the
functionality, it is stays as `pub(crate)`.

The threaded scheduler is now exposed under `rt-threaded`. This feature
flag only enables the threaded scheduler and does not include I/O,
networking, or time. Those features must be explictly enabled.

A `full` feature flag is added that enables all features.

`stdin`, `stdout`, `stderr` are exposed under `io-std`.

Macros are used to scope code by feature flag.
2019-11-18 07:00:55 -08:00
sclaire-1 13b6e9939e Edit CONTRIBUTING.md (#1784)
Edited the last sentence of the first section to improve clarity
2019-11-17 23:27:42 -08:00
Carl Lerche 44f10fe47f sync: require T: Clone for watch channels. (#1783)
There are limitations with `async/await` (no GAT) requiring the value to
be cloned on receive. The `poll` based API is not currently exposed.
This makes the `Clone` requirement explicit.
2019-11-17 09:03:44 -08:00
Carl Lerche c147be0437 make AtomicWaker private (#1782) 2019-11-16 23:35:17 -08:00
Carl Lerche b1d9e55487 task: move blocking fns into tokio::task (#1781) 2019-11-16 23:35:04 -08:00
Taiki Endo 66cbed3ce3 tls: enable test on CI (#1779) 2019-11-16 22:24:58 -08:00
Carl Lerche 4d19a99937 runtime: set spawn context on enter (#1780) 2019-11-16 22:24:28 -08:00
Taiki Endo 10dc659450 io: expose std{in, out, err} under io feature (#1759)
This exposes `std{in, out, err}` under io feature by moving
`fs::blocking` module into `io::blocking`.
As `fs` feature depends on `io-trait` feature, `fs` implementations can
always access `io` module.
2019-11-16 22:03:39 -08:00
Taiki Endo 320c84a433 chore: migrate from pin-project to pin-project-lite (#1778) 2019-11-16 09:14:40 -08:00
Carl Lerche 19f1fc36bd task: return JoinHandle from spawn (#1777)
`tokio::spawn` now returns a `JoinHandle` to obtain the result of the task:

Closes #887.
2019-11-16 08:28:34 -08:00
Carl Lerche 3f0eabe779 runtime: rename current_thread -> basic_scheduler (#1769)
It no longer supports executing !Send futures. The use case for
It is wanting a “light” runtime. There will be “local” task execution
using a different strategy coming later.

This patch also renames `thread_pool` -> `threaded_scheduler`, but
only in public APIs for now.
2019-11-16 07:19:45 -08:00
Taiki Endo 1474794055 runtime: allow non-unit type output in {Runtime, Spawner}::spawn (#1756) 2019-11-15 22:16:21 -08:00
Taiki Endo 92eb635669 net: add more impls for ToSocketAddrs (#1760) 2019-11-15 22:12:57 -08:00
Carl Lerche 8a7e57786a Limit futures dependency to Stream via feature flag (#1774)
In an effort to reach API stability, the `tokio` crate is shedding its
_public_ dependencies on crates that are either a) do not provide a
stable (1.0+) release with longevity guarantees or b) match the `tokio`
release cadence. Of course, implementing `std` traits fits the
requirements.

The on exception, for now, is the `Stream` trait found in `futures_core`.
It is expected that this trait will not change much and be moved into `std.
Since Tokio is not yet going reaching 1.0, I feel that it is acceptable to maintain
a dependency on this trait given how foundational it is.

Since the `Stream` implementation is optional, types that are logically
streams provide `async fn next_*` functions to obtain the next value.
Avoiding the `next()` name prevents fn conflicts with `StreamExt::next()`.

Additionally, some misc cleanup is also done:

- `tokio::io::io` -> `tokio::io::util`.
- `delay` -> `delay_until`.
- `Timeout::new` -> `timeout(...)`.
- `signal::ctrl_c()` returns a future instead of a stream.
- `{tcp,unix}::Incoming` is removed (due to lack of `Stream` trait).
- `time::Throttle` is removed (due to lack of `Stream` trait).
-  Fix: `mpsc::UnboundedSender::send(&self)` (no more conflict with `Sink` fns).
2019-11-15 22:11:13 -08:00
Markus Westerlind 930679587a codec: Remove Unpin requirement from Framed[Read,Write,] (#1758)
cc #1252
2019-11-15 16:30:07 +09:00
Carl Lerche 27e5b41067 reorganize modules (#1766)
This patch started as an effort to make `time::Timer` private. However, in an
effort to get the build compiling again, more and more changes were made. This
probably should have been broken up, but here we are. I will attempt to
summarize the changes here.

* Feature flags are reorganized to make clearer. `net-driver` becomes
  `io-driver`. `rt-current-thread` becomes `rt-core`.

* The `Runtime` can be created without any executor. This replaces `enter`. It
  also allows creating I/O / time drivers that are standalone.

* `tokio::timer` is renamed to `tokio::time`. This brings it in line with `std`.

* `tokio::timer::Timer` is renamed to `Driver` and made private.

* The `clock` module is removed. Instead, an `Instant` type is provided. This
  type defaults to calling `std::time::Instant`. A `test-util` feature flag can
  be used to enable hooking into time.

* The `blocking` module is moved to the top level and is cleaned up.

* The `task` module is moved to the top level.

* The thread-pool's in-place blocking implementation is cleaned up.

* `runtime::Spawner` is renamed to `runtime::Handle` and can be used to "enter"
  a runtime context.
2019-11-12 15:23:40 -08:00
Anton Barkovsky e3df2eafd3 tls: fix test certificate to work on macOS 10.15 (#1763)
macOS 10.15 introduced new requirements for certificates to be trusted:
https://support.apple.com/en-us/HT210176
2019-11-11 12:09:14 +01:00
Taiki Endo c15e01a09b chore: remove rust-toolchain and add minimum supported version check (#1748)
* remove rust-toolchain

* add minimum supported version check
2019-11-08 13:26:08 +09:00
Taiki Endo 64f2bf0072 chore: update CI config to test on stable (#1747) 2019-11-08 00:32:04 +09:00
Carl Lerche 7e35922a1d time: rename tokio::timer -> tokio::time (#1745) 2019-11-06 23:53:46 -08:00
Carl Lerche 4dbe6af0a1 runtime: misc pool cleanup (#1743)
- Remove builders for internal types
- Avoid duplicating the blocking pool when using the concurrent
  scheduler.
- misc smaller cleanup
2019-11-06 21:29:10 -08:00
leo-lb 9bec094150 timer: have example use delay_for instead of delay (#1735)
It is a more common use case that is to simply cause a delay for an amount of time.
I think it is more appropriate to show off `delay_for` in the example rather than `delay` that is useful only for less common use cases.
2019-11-06 21:28:21 -08:00
Taiki Endo 6f8b986bdb chore: update futures to 0.3.0 (#1741) 2019-11-07 05:09:10 +09:00
Carl Lerche 1a7f6fb201 simplify enter (#1736) 2019-11-06 09:51:15 -08:00
Carl Lerche 0da23aad77 fix clippy (#1737) 2019-11-05 23:38:52 -08:00
Carl Lerche d5c1119c88 runtime: combine executor and runtime mods (#1734)
Now, all types are under `runtime`. `executor::util` is moved to a top
level `util` module.
2019-11-05 19:12:30 -08:00
Carl Lerche a6253ed05a chore: unify all mocked loom files (#1732)
When the crates were merged, each component kept its own `loom` file
containing mocked types it needed. This patch unifies them all in one
location.
2019-11-04 22:22:40 -08:00
Carl Lerche 94f9b04b06 executor: switch some APIs to crate private. (#1731)
* switch `enter` to crate private
* make executor types pub(crate)
2019-11-04 14:12:24 -08:00
Carl Lerche 966ccd5d53 test: unify MockTask and task::spawn (#1728)
Delete `MockTask` in favor of `task::spawn`. Both are functionally
equivalent.
2019-11-03 14:10:14 -08:00
Taiki Endo 3948e16292 ci: install minimal profile by default (#1729) 2019-11-03 12:08:07 -08:00
Sebastian Dröge 6b35a1e8b0 impl AsyncWrite for std::io::Cursor (#1730)
Based on the implementation from the futures crate.
2019-11-03 21:21:01 +09:00
Carl Lerche e19bd77ef0 tests: fix bug + reorganize tests. (#1726)
Fixes a bug in the thread-pool executor related to shutdown
concurrent with a task that is self-notifying. A `loom` test is
added to validate the fix.

Additionally, in anticipation of the `thread_pool` module being
switched to private, tests are updated to use `Runtime` directly
instead of `thread_pool`. Those tests that cannot be updated
are switched to unit tests.
2019-11-02 17:03:06 -07:00
Carl Lerche c8fdbed27a chore: prune dev-dependencies
Most dev dependendencies are unused now that examples are moved into a
separate crate.
2019-11-02 09:40:37 +01:00
Carl Lerche 3e7d0be51d executor: remove Executor & TypedExecutor traits (#1724)
The `Executor` trait is sub-optimal as it forces a `Box<dyn Future>` to
spawn. Instead, `tokio::spawn` delegates to the specific runtime
implementation set for the current execution context.

`TypedExecutor`, while useful, has seen limited adoption. As such, it is
removed from `tokio` proper. Moving it to `tokio-util` is a possibility
that can be explored as follow up work.
2019-11-01 13:50:17 -07:00
Carl Lerche d70c928d88 runtime: merge multi & single threaded runtimes (#1716)
Simplify Tokio's runtime construct by combining both Runtime variants
into a single type. The execution style can be controlled by a
configuration setting on `Builder`.

The implication of this change is that there is no longer any way to
spawn `!Send` futures. This, however, is a temporary limitation. A
different strategy will be employed for supporting `!Send` futures.

Included in this patch is a rework of `task::JoinHandle` to support
using this type from both the thread-pool and current-thread executors.
2019-11-01 13:18:52 -07:00
Steven Fackler 742d89b0f3 Fix delay construction from non-lazy Handles (#1720)
Closes #1719.
2019-11-01 12:32:57 -07:00
Carl Lerche 20993341bd compat: extract crate to a dedicated git repo (#1723)
The compat crate is moved to https://github.com/tokio-rs/tokio-compat.
This allows pinning it to specific revisions of the Tokio git
repository. The master branch is intended to go through significant
churn and it will be easier to update the compat layer in batches.
2019-11-01 12:30:12 -07:00
Eliza Weisman e699d46534 compat: add a compat runtime (#1663)
## Motivation

The `futures` crate's [`compat` module][futures-compat] provides
interoperability between `futures` 0.1 and `std::future` _future types_
(e.g. implementing `std::future::Future` for a type that implements the
`futures` 0.1 `Future` trait). However, this on its own is insufficient
to run code written against `tokio` 0.1 on a `tokio` 0.2 runtime, if
that code also relies on `tokio`'s runtime services. If legacy tasks are
executed that rely on `tokio::timer`, perform IO using `tokio`'s
reactor, or call `tokio::spawn`, those API calls will fail unless there
is also a runtime compatibility layer.

## Solution

As proposed in #1549, this branch introduces a new `tokio-compat` crate,
with implementations of the thread pool and current-thread runtimes that
are capable of running both tokio 0.1 and tokio 0.2 tasks. The compat
runtime creates a background thread that runs a `tokio` 0.1 timer and
reactor, and sets itself as the `tokio` 0.1 executor as well as the
default 0.2 executor. This allows 0.1 futures that use 0.1 timer,
reactor, and executor APIs may run alongside `std::future` tasks on the
0.2 runtime.

### Examples

Spawning both `tokio` 0.1 and `tokio` 0.2 futures:

```rust
use futures_01::future::lazy;

tokio_compat::run(lazy(|| {
    // spawn a `futures` 0.1 future using the `spawn` function from the
    // `tokio` 0.1 crate:
    tokio_01::spawn(lazy(|| {
        println!("hello from tokio 0.1!");
        Ok(())
    }));

    // spawn an `async` block future on the same runtime using `tokio`
    // 0.2's `spawn`:
    tokio_02::spawn(async {
        println!("hello from tokio 0.2!");
    });

    Ok(())
}))
```

Futures on the compat runtime can use `timer` APIs from both 0.1 and 0.2
versions of `tokio`:

```rust
use std::time::{Duration, Instant};
use futures_01::future::lazy;
use tokio_compat::prelude::*;

tokio_compat::run_03(async {
    // Wait for a `tokio` 0.1 `Delay`...
    let when = Instant::now() + Duration::from_millis(10);
    tokio_01::timer::Delay::new(when)
        // convert the delay future into a `std::future` that we can `await`.
        .compat()
        .await
        .expect("tokio 0.1 timer should work!");
    println!("10 ms have elapsed");

    // Wait for a `tokio` 0.2 `Delay`...
    let when = Instant::now() + Duration::from_millis(20);
    tokio_02::timer::delay(when).await;
    println!("20 ms have elapsed");
});
```

## Future Work

This is just an initial implementation of a `tokio-compat` crate; there
are more compatibility layers we'll want to provide before that crate is
complete. For example, we should also provide compatibility between
`tokio` 0.2's `AsyncRead` and `AsyncWrite` traits and the `futures` 0.1
and `futures` 0.3 versions of those traits. In #1549, @carllerche also
suggests that the `compat` crate provide reimplementations of APIs that
were removed from `tokio` 0.2 proper, such as the `tcp::Incoming`
future.

Additionally, there is likely extra work required to get the 
`tokio-threadpool` 0.1 `blocking` APIs to work on the compat runtime.
This will be addressed in a follow-up PR.

Fixes: #1605
Fixes: #1552
Refs: #1549

[futures-compat]: https://rust-lang-nursery.github.io/futures-api-docs/0.3.0-alpha.19/futures/compat/index.html
2019-11-01 10:35:02 -07:00
Carl Lerche 72caede7be chore: remove dead files (#1718)
The `codec` module has been moved to `tokio-util`. Some files were left,
but they were never activated.
2019-11-01 21:30:06 +09:00
Carl Lerche 64c26ab1ee runtime: test creating a single-threaded runtime. (#1717) 2019-10-31 22:28:31 -07:00
Taiki Endo 02f7264008 chore: check each feature works properly (#1695)
It is hard to maintain features list manually, so use cargo-hack's
`--each-feature` flag. And cargo-hack provides a workaround for an issue
that dev-dependencies leaking into normal build (`--no-dev-deps` flag),
so removed own ci tool.

Also, compared to running tests on all features, there is not much
advantage in running tests on each feature, so only the default features
and all features are tested.
If the behavior changes depending on the feature, we need to test it as
another job in CI.
2019-10-31 21:09:32 -07:00
Jonathan Bastien-Filiatrault 2902e39db0 Allow non-destructive access to the read buffer. (#1600)
I need this to implement SMTP pipelining checks. I mostly need to
flush my send buffer when the read buffer is empty before waiting for
the next command.
2019-10-31 10:36:24 -04:00
Steven Fackler 630d3136dd timere: make Delay must_use (#1714)
Closes #1711
2019-10-30 20:21:03 -07:00
Sean McArthur 2c870b588f process: refactor OrphanQueue to use a Mutex instead fo SegQueue (#1712) 2019-10-30 15:29:04 -07:00
Jon Gjengset 109fd3086b thread-pool: in-place blocking with new scheduler (#1681)
The initial new scheduler PR omitted in-place blocking
support. This patch brings it back.
2019-10-30 08:58:49 -07:00
Sean McArthur e3261440e5 timer: inline CachePadded type (#1706) 2019-10-29 22:16:11 -07:00
Carl Lerche 2b909d6805 sync: move into tokio crate (#1705)
A step towards collapsing Tokio sub crates into a single `tokio`
crate (#1318).

The sync implementation is now provided by the main `tokio` crate.
Functionality can be opted out of by using the various net related
feature flags.
2019-10-29 15:11:31 -07:00
Carl Lerche c62ef2d232 executor: move into tokio crate (#1702)
A step towards collapsing Tokio sub crates into a single `tokio`
crate (#1318).

The executor implementation is now provided by the main `tokio` crate.
Functionality can be opted out of by using the various net related
feature flags.
2019-10-28 21:40:29 -07:00
Eliza Weisman 7eb264a0d0 net: replace RwLock<Slab> with a lock free slab (#1625)
## Motivation

The `tokio_net::driver` module currently stores the state associated
with scheduled IO resources in a `Slab` implementation from the `slab`
crate. Because inserting items into and removing items from `slab::Slab`
requires mutable access, the slab must be placed within a `RwLock`. This
has the potential to be a performance bottleneck especially in the context of
the work-stealing scheduler where tasks and the reactor are often located on
the same thread.

`tokio-net` currently reimplements the `ShardedRwLock` type from
`crossbeam` on top of `parking_lot`'s `RwLock` in an attempt to squeeze
as much performance as possible out of the read-write lock around the
slab. This introduces several dependencies that are not used elsewhere.

## Solution

This branch replaces the `RwLock<Slab>` with a lock-free sharded slab
implementation. 

The sharded slab is based on the concept of _free list sharding_
described by Leijen, Zorn, and de Moura in [_Mimalloc: Free List
Sharding in Action_][mimalloc], which describes the implementation of a
concurrent memory allocator. In this approach, the slab is sharded so
that each thread has its own thread-local list of slab _pages_. Objects
are always inserted into the local slab of the thread where the
insertion is performed. Therefore, the insert operation needs not be
synchronized.

However, since objects can be _removed_ from the slab by threads other
than the one on which they were inserted, removal operations can still
occur concurrently. Therefore, Leijen et al. introduce a concept of
_local_ and _global_ free lists. When an object is removed on the same
thread it was originally inserted on, it is placed on the local free
list; if it is removed on another thread, it goes on the global free
list for the heap of the thread from which it originated. To find a free
slot to insert into, the local free list is used first; if it is empty,
the entire global free list is popped onto the local free list. Since
the local free list is only ever accessed by the thread it belongs to,
it does not require synchronization at all, and because the global free
list is popped from infrequently, the cost of synchronization has a
reduced impact. A majority of insertions can occur without any
synchronization at all; and removals only require synchronization when
an object has left its parent thread.

The sharded slab was initially implemented in a separate crate (soon to
be released), vendored in-tree to decrease `tokio-net`'s dependencies.
Some code from the original implementation was removed or simplified,
since it is only necessary to support `tokio-net`'s use case, rather
than to provide a fully generic implementation.

[mimalloc]: https://www.microsoft.com/en-us/research/uploads/prod/2019/06/mimalloc-tr-v1.pdf

## Performance

These graphs were produced by out-of-tree `criterion` benchmarks of the
sharded slab implementation.


The first shows the results of a benchmark where an increasing number of
items are inserted and then removed into a slab concurrently by five
threads. It compares the performance of the sharded slab implementation
with a `RwLock<slab::Slab>`:

<img width="1124" alt="Screen Shot 2019-10-01 at 5 09 49 PM" src="https://user-images.githubusercontent.com/2796466/66078398-cd6c9f80-e516-11e9-9923-0ed6292e8498.png">

The second graph shows the results of a benchmark where an increasing
number of items are inserted and then removed by a _single_ thread. It
compares the performance of the sharded slab implementation with an
`RwLock<slab::Slab>` and a `mut slab::Slab`.

<img width="925" alt="Screen Shot 2019-10-01 at 5 13 45 PM" src="https://user-images.githubusercontent.com/2796466/66078469-f0974f00-e516-11e9-95b5-f65f0aa7e494.png">

Note that while the `mut slab::Slab` (i.e. no read-write lock) is
(unsurprisingly) faster than the sharded slab in the single-threaded
benchmark, the sharded slab outperforms the un-contended
`RwLock<slab::Slab>`. This case, where the lock is uncontended and only
accessed from a single thread, represents the best case for the current
use of `slab` in `tokio-net`, since the lock cannot be conditionally
removed in the single-threaded case.

These benchmarks demonstrate that, while the sharded approach introduces
a small constant-factor overhead, it offers significantly better
performance across concurrent accesses.

## Notes

This branch removes the following dependencies `tokio-net`:
- `parking_lot`
- `num_cpus`
- `crossbeam_util`
- `slab`

This branch adds the following dev-dependencies:
- `proptest`
- `loom`

Note that these dev dependencies were used to implement tests for the
sharded-slab crate out-of-tree, and were necessary in order to vendor
the existing tests. Alternatively, since the implementation is tested
externally, we _could_ remove these tests in order to avoid picking up
dev-dependencies. However, this means that we should try to ensure that
`tokio-net`'s vendored implementation doesn't diverge significantly from
upstream's, since it would be missing a majority of its tests.

Signed-off-by: Eliza Weisman <[email protected]>
2019-10-28 11:30:45 -07:00
Geoff Shannon 1195263584 Fix docs links: Redux (#1698) 2019-10-27 09:37:07 -07:00
Carl Lerche bccb713d98 thread-pool: test additional shutdown cases (#1697)
This adds an extra spawned task during the thread-pool shutdown loom
test. This results in additional cases being tested, primarily tasks
being stolen.
2019-10-26 22:15:39 -07:00
Linus Färnstrand 474befd23c chore: use argument position impl trait (#1690) 2019-10-26 08:40:38 -07:00
Carl Lerche 987ba7373c io: move into tokio crate (#1691)
A step towards collapsing Tokio sub crates into a single `tokio`
crate (#1318).

The `io` implementation is now provided by the main `tokio` crate.
Functionality can be opted out of by using the various net related
feature flags.
2019-10-26 08:02:49 -07:00
Carl Lerche 227533d456 net: move into tokio crate (#1683)
A step towards collapsing Tokio sub crates into a single `tokio`
crate (#1318).

The `net` implementation is now provided by the main `tokio` crate.
Functionality can be opted out of by using the various net related
feature flags.
2019-10-25 12:50:15 -07:00
Jon Gjengset 03a9378297 Make blocking pool non-static and use for thread pool (#1678)
Previously, support for `blocking` was done through a static `POOL` that
would spawn threads on demand. While this made the pool accessible at
all times, it made it hard to configure, and it was impossible to keep
multiple blocking pools.

This patch changes `blocking` to instead use a "default" global like the
ones used for timers, executors, and the like. There is now
`blocking::with_pool`, which is used by both thread-pool workers and the
current-thread runtime to ensure that a pool is available to tasks.

This patch also changes `ThreadPool` to spawn its worker threads on the
blocking pool rather than as free-standing threads. This is in
preparation for the coming in-place blocking work.

One downside of this change is that thread names are no longer
"semantic". All threads are named by the pool name, and individual
threads are not (currently) given names with numerical suffixes like
before.
2019-10-24 14:17:47 -07:00
Carl Lerche 99940aeeb4 chore: remove tracing. (#1680)
Historically, logging has been added haphazardly. Here, we entirely
remove logging as none of it is particularly useful. In the future, we
will add tracing back in order to expose useful data to the user of
Tokio.
2019-10-23 11:04:14 -07:00
Carl Lerche cfc15617a5 codec: move into tokio-util (#1675)
Related to #1318, Tokio APIs that are "less stable" are moved into a new
`tokio-util` crate. This crate will mirror `tokio` and provide
additional APIs that may require a greater rate of breaking changes.

As examples require `tokio-util`, they are moved into a separate
crate (`examples`). This has the added advantage of being able to avoid
example only dependencies in the `tokio` crate.
2019-10-22 10:13:49 -07:00
Carl Lerche b8cee1a60a timer: move tokio-timer into tokio crate (#1674)
A step towards collapsing Tokio sub crates into a single `tokio`
crate (#1318).

The `timer` implementation is now provided by the main `tokio` crate.
The `timer` functionality may still be excluded from the build by
skipping the `timer` feature flag.
2019-10-21 16:45:13 -07:00
Kevin Leimkuhler c9bcbe77b9 net: Eagerly bind resources to reactors (#1666)
## Motivation

The `tokio_net` resources can be created outside of a runtime due to how tokio
has been used with futures to date. For example, this allows a `TcpStream` to be
created, and later passed into a runtime:

```
let stream = TcpStream::connect(...).and_then(|socket| {
    // do something
});
tokio::run(stream);
```

In order to support this functionality, the reactor was lazily bound to the
resource on the first call to `poll_read_ready`/`poll_write_ready`. This
required a lot of additional complexity in the binding logic to support.

With the tokio 0.2 common case, this is no longer necessary and can be removed.
All resources are expected to be created from within a runtime, and should panic
if not done so.

Closes #1168

## Solution

The `tokio_net` crate now assumes there to be a `CURRENT_REACTOR` set on the
worker thread creating a resource; this can be assumed if called within a tokio
runtime. If there is no current reactor, the application will panic with a "no
current reactor" message.

With this assumption, all the unsafe and atomics have been removed from
`tokio_net::driver::Registration` as it is no longer needed.

There is no longer any reason to pass in handles to the family of `from_std` methods on `net` resources. `Handle::current` has therefore a more restricted private use where it is only used in `driver::Registration::new`.

Signed-off-by: Kevin Leimkuhler <[email protected]>
2019-10-21 16:20:06 -07:00
Carl Lerche 978013a215 fs: move into tokio (#1672)
A step towards collapsing Tokio sub crates into a single `tokio`
crate (#1318).

The `fs` implementation is now provided by the main `tokio` crate. The
`fs` functionality may still be excluded from the build by skipping the
`fs` feature flag.
2019-10-21 15:49:00 -07:00
madmaxio 6aa6ebb5bc io: Take struct re-export to main crate (#1670) 2019-10-21 10:03:05 -07:00
Jonathas Conceição 4bee94eb06 runtime: update doc regarding runtime::run function helper (#1671) 2019-10-21 10:02:36 -07:00
Carl Lerche ed5a94eb2d executor: rewrite the work-stealing thread pool (#1657)
This patch is a ground up rewrite of the existing work-stealing thread
pool. The goal is to reduce overhead while simplifying code when
possible.

At a high level, the following architectural changes were made:

- The local run queues were switched for bounded circle buffer queues.
- Reduce cross-thread synchronization.
- Refactor task constructs to use a single allocation and always include
  a join handle (#887).
- Simplify logic around putting workers to sleep and waking them up.

**Local run queues**

Move away from crossbeam's implementation of the Chase-Lev deque. This
implementation included unnecessary overhead as it supported
capabilities that are not needed for the work-stealing thread pool.
Instead, a fixed size circle buffer is used for the local queue. When
the local queue is full, half of the tasks contained in it are moved to
the global run queue.

**Reduce cross-thread synchronization**

This is done via many small improvements. Primarily, an upper bound is
placed on the number of concurrent stealers. Limiting the number of
stealers results in lower contention. Secondly, the rate at which
workers are notified and woken up is throttled. This also reduces
contention by preventing many threads from racing to steal work.

**Refactor task structure**

Now that Tokio is able to target a rust version that supports
`std::alloc` as well as `std::task`, the pool is able to optimize how
the task structure is laid out. Now, a single allocation per task is
required and a join handle is always provided enabling the spawner to
retrieve the result of the task (#887).

**Simplifying logic**

When possible, complexity is reduced in the implementation. This is done
by using locks and other simpler constructs in cold paths. The set of
sleeping workers is now represented as a `Mutex<VecDeque<usize>>`.
Instead of optimizing access to this structure, we reduce the amount the
pool must access this structure.

Secondly, we have (temporarily) removed `threadpool::blocking`. This
capability will come back later, but the original implementation was way
more complicated than necessary.

**Results**

The thread pool benchmarks have improved significantly:

Old thread pool:

```
test chained_spawn ... bench:   2,019,796 ns/iter (+/- 302,168)
test ping_pong     ... bench:   1,279,948 ns/iter (+/- 154,365)
test spawn_many    ... bench:  10,283,608 ns/iter (+/- 1,284,275)
test yield_many    ... bench:  21,450,748 ns/iter (+/- 1,201,337)
```

New thread pool:

```
test chained_spawn ... bench:     147,943 ns/iter (+/- 6,673)
test ping_pong     ... bench:     537,744 ns/iter (+/- 20,928)
test spawn_many    ... bench:   7,454,898 ns/iter (+/- 283,449)
test yield_many    ... bench:  16,771,113 ns/iter (+/- 733,424)
```

Real-world benchmarks improve significantly as well. This is testing the hyper hello
world server using: `wrk -t1 -c50 -d10`:

Old scheduler:

```
Running 10s test @ http://127.0.0.1:3000
  1 threads and 50 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   371.53us   99.05us   1.97ms   60.53%
    Req/Sec   114.61k     8.45k  133.85k    67.00%
  1139307 requests in 10.00s, 95.61MB read
Requests/sec: 113923.19
Transfer/sec:      9.56MB
```

New scheduler:

```
Running 10s test @ http://127.0.0.1:3000
  1 threads and 50 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   275.05us   69.81us   1.09ms   73.57%
    Req/Sec   153.17k    10.68k  171.51k    71.00%
  1522671 requests in 10.00s, 127.79MB read
Requests/sec: 152258.70
Transfer/sec:     12.78MB
```
2019-10-19 11:09:40 -07:00
Steven Fackler 2a181320b7 fs: add read_to_string (#1664) 2019-10-16 15:47:37 -07:00
Taiki Endo 4c97e9dc28 fs: remove unnecessary trait and lifetime bounds (#1655) 2019-10-15 19:02:34 +09:00
Jon Gjengset 1cae04f8b3 macros: Use more consistent runtime names (#1628)
As discussed in #1620, the attribute names for `#[tokio::main]` and
`#[tokio::test]` aren't great. Specifically, they both use
`single_thread` and `multi_thread`, as opposed to names that match the
runtime names: `current_thread` and `threadpool`. This PR changes the
former to the latter.

Fixes #1627.
2019-10-12 12:55:39 -04:00
John-John Tedro 29f35df7f8 Remove incorrect FusedFuture impl on Delay (#1652)
`is_terminated` must return `true` until the future has been polled at least once to make sure that the associated block in select is called even after the delay has elapsed.

You use `Delay` in a `select!` by [fusing it](https://docs.rs/futures-preview/0.3.0-alpha.19/futures/future/trait.FutureExt.html#method.fuse):

```rust
let delay = tokio::timer::delay(/* ... */);
let delay = delay.fuse();

select! {
    _ = delay => {
        /* work here */
    }
}
```
2019-10-11 15:45:44 -04:00
Ivan Petkov 741bef8fe1 tokio: move signal and process reexports to crate root (#1643) 2019-10-11 11:00:39 -07:00
Carl Lerche 804dbd6f8e sync: fix mem leak in oneshot on task migration (#1648)
When polling the task, the current waker is saved to the oneshot state.
When the handle is migrated to a new task and polled again, the waker
must be swaped from the old waker to the new waker. In some cases, there
is a potential for the old waker to leak.

This bug was caught by loom with the recently added memory leak
detection.
2019-10-10 12:00:22 -07:00
Eliza Weisman 69fe65e972 io: add AsyncBufReadExt::split (#1642)
add a `split` method to `AsyncBufReadExt`, analogous to `std::io::BufRead::split`.
2019-10-09 13:17:07 -07:00
Jonathan Bastien-Filiatrault b8913ec7c0 executor: accurate idle thread tracking for the blocking pool (#1621)
Use a counter to count notifications. This protects against spurious
wakeups by pthreads and other libraries. The state transitions now
track num_idle precisely.
2019-10-07 14:04:28 -07:00
Eliza Weisman 8aa520e2bd io: add missing utility functions (#1632)
The standard library's `io` module has small utilities such as `repeat`,
`empty`, and `sink`, which return `Read` and `Write` implementations.
These can come in handy in some circiumstances. `tokio::io` has no
equivalents that implement `AsyncRead`/`AsyncWrite`.

This commit adds `repeat`, `empty`, and `sink` helpers to `tokio::io`.
2019-10-07 14:02:04 -07:00
Nick Stott ab2f71a612 chore: fix a comment typo (#1633) 2019-10-07 09:20:57 -07:00
Taiki Endo 42a5cb1508 timer: test arm on targets with target_has_atomic less than 64 (#1634) 2019-10-07 09:19:44 -07:00
Taiki Endo 2b4b0619d7 chore: update Cirrus CI config to test on beta (#1636) 2019-10-07 09:18:38 -07:00
Taiki Endo 55caddb9ce chore: do not trigger CI on std-future branch (#1635) 2019-10-07 09:17:27 -07:00
Vojtech Kral aefaef3abf tcp: export Incoming type (#1602) 2019-10-02 11:12:05 -07:00
Jon Gjengset c78c9168d7 macros: allow selecting runtime in tokio::test attr (#1620)
In the past, it was not possible to choose to use the multi-threaded
tokio `Runtime` in tests, which meant that any test that transitively
used `executor::threadpool::blocking` would fail with

```
'blocking' annotation used from outside the context of a thread pool
```

This patch adds a runtime annotation attribute to `#[tokio::test]` just
like `#[tokio::main]` has, which lets users opt in to the threadpool
runtime over `current_thread` (the default).
2019-10-02 10:58:34 -07:00
Jonathan Bastien-Filiatrault 9e1eef829a chore: annotate prelude re-exports as doc(no_inline) (#1601)
Fixes #1593 by making "use as _" linked in the documentation.
2019-10-02 10:55:35 -07:00
Taiki Endo f48980ae52 chore: update rust-toolchain to use beta (#1619) 2019-10-01 10:13:38 -04:00
Douman a1d1eb5eb3 macros: Allow arguments in non-main functions 2019-10-01 13:15:46 +02:00
Jon Gjengset 5efe31f2ed Prepare for release of 0.2.0-alpha.6 (#1617)
Note that `tokio-timer` and `tokio-tls` become 0.3.0-alpha.6 (not 0.2.0)
2019-09-30 18:35:52 -04:00
Jon Gjengset 5ce5a0a0e0 Fix for rust-lang/rust#64477 (#1618)
`foo(format!(...)).await` no longer compiles. There's a fix in
rust-lang/rust#64856, but this works around the problem.
2019-09-30 17:17:14 -04:00
Jon Gjengset 5fd5329497 Create BufStream from a BufReader + BufWriter (#1609)
This is handy if developers want to construct the inner buffers with a
particular capacity, and still end up with a `BufStream` at the end.
2019-09-30 14:22:59 -04:00
Taiki Endo 3b8ee2d991 chore: update futures-preview to 0.3.0-alpha.19 (#1610) 2019-09-30 13:32:37 -04:00
Jon Gjengset 7c341f45e0 chore: move CI to beta (#1615) 2019-09-27 09:51:45 -07:00
Jon Gjengset 611b4e11a7 Make Barrier::wait future Send (#1611)
It wasn't before. Now it is. And that is better.
2019-09-26 18:26:24 -04:00
Taiki Endo 159abb375f chore: update pin-project to 0.4 (#1603) 2019-09-27 04:51:28 +09:00
Carl Lerche 032b39487c sync: add spin_loop_hint to atomic waker (#1608)
The algorithm backing `AtomicWaker` effectively uses a spin lock backed
by notifying & yielding the current task. This adds a `spin_lock_hint`
annotation to cover this case.

While, in practice, the omission of `spin_lock_hint` would not cause
problems, there are platforms that do not handle spin locks very well
and could enter a deadlock in pathological cases.
2019-09-26 15:16:34 -04:00
Hung-I Wang b71b7b36be fs: update the doc comment of File::sync_data (#1596) 2019-09-25 09:05:50 -07:00
Taiki Endo c4567f741a io: add get_*/into_inner methods to BufStream (#1598) 2019-09-25 09:17:43 -04:00
Sean McArthur 18cef1901f tokio: add rt-current-thread optional feature
- Adds a minimum `rt-current-thread` optional feature that exports
  `tokio::runtime::current_thread`.
- Adds a `macros` optional feature to enable the `#[tokio::main]` and
  `#[tokio::test]` attributes.
- Adjusts `#[tokio::main]` macro to select a runtime "automatically" if
  a specific strategy isn't specified. Allows using the macro with only
  the rt-current-thread feature.
2019-09-24 12:17:04 -07:00
Taiki Endo c81447fdcc io: remove unsafe pin-projections and remove manual Unpin implementations (#1588)
* Removes most pin-projection related unsafe code.

* Removes manual Unpin implementations.
  As references always implement Unpin, there is no need to implement
  Unpin manually.

* Adds tests to check that Unpin requirement does not change accidentally 
  because changing Unpin requirements will be breaking changes.
2019-09-25 01:17:06 +09:00
Taiki Endo d50d050fae net: fix build-tests for uds (#1589) 2019-09-24 02:47:39 +09:00
Taiki Endo 3a55aba251 macros: add build tests for #[tokio::main] and #[tokio::test] (#1591) 2019-09-23 04:09:30 +09:00
Taiki Endo ddbb0c3836 macros: fix handling of arguments of #[tokio::main] attribute (#1578) 2019-09-23 03:05:04 +09:00
Taiki Endo 376d63867a chore: update pin-project to 0.4.0-beta.1 (#1586) 2019-09-23 01:52:14 +09:00
Taiki Endo eb2d0fbcd1 net: use Box::pin instead of Pin::new(Box::new) (#1587) 2019-09-22 09:28:55 -07:00
Jonathan Bastien-Filiatrault 695165feac timer: 32 bit ARM only has 32 bit atomics. (#1581) 2019-09-20 13:25:49 -07:00
Kirill Mironov ff186a4d03 tokio: add process feature (#1561) 2019-09-19 19:03:58 -07:00
Jon Gjengset 6611b32cce Export sync::Barrier from tokio::sync (#1577) 2019-09-19 21:32:35 -04:00
Carl Lerche 80ba2a4ff6 Release 0.2.0 alpha.5 (#1576) 2019-09-19 13:39:35 -07:00
Carl Lerche 8d09f61d33 net: fix build with only process (#1575) 2019-09-19 12:38:15 -07:00
Carl Lerche 815173f8e5 chore: rm tokio-buf (#1574)
The crate has not been updated and it does not seem like it is a good
path forward.
2019-09-19 12:11:21 -07:00
Markus Westerlind 34e388619f timer: delay_for should use tokio_timer::clock::now (#1572) 2019-09-19 11:20:18 -07:00
Jon Gjengset 9d5af20bcf Enable buffering both reads and writes (#1558)
`BufWriter` and `BufReader` did not previously forward the "opposite" trait (`AsyncRead` for `BufWriter` and `AsyncWrite` for `BufReader`). This meant that there was no way to have both directions buffered at once. This patch fixes that, and introduces a convenience type + constructor for this double-wrapped construct.
2019-09-19 14:17:15 -04:00
Jon Gjengset 613fde2637 sync: add Barrier primitive (#1571)
This adds `Barrier` to `tokio-sync`, which is an asynchronous alternative to [`std::sync::Barrier`](https://doc.rust-lang.org/std/sync/struct.Barrier.html). It is a synchronization primitive that allows multiple futures to "rendezvous" at certain points in their execution.
2019-09-19 14:16:56 -04:00
Jonathan Bastien-Filiatrault 22a3b10171 executor: fix blocking pool bug re: thread shutdown (#1562)
Currently, when threads in the blocking pool shutdown due to being idle
the counter tracking threads is not decremented. This prevents new threads
from being spawned to replace the shutdown threads.
2019-09-19 11:12:04 -07:00
Jon Gjengset e3415d8d61 sync: Make Lock more similar to std::sync::Mutex (#1573)
This renames `Lock` to `Mutex`, and brings the API more in line with `std::sync::Mutex`.

In partcular, locking now only takes `&self`, with the expectation that you place the `Mutex` in an `Arc` (or something similar) to share it between threads.

Fixes #1544.
Part of #1210.
2019-09-19 11:46:52 -04:00
Taiki Endo d1f60ac4c6 chore: deny warnings for doc tests (#1539) 2019-09-19 15:50:12 +09:00
Taiki Endo e2161502ad chore: fix clippy check failure (#1569) 2019-09-18 10:19:44 -07:00
yjh ab785bfba7 Update README.md (#1545)
change url's `version` to `latest`.
2019-09-17 10:54:22 -04:00
Lucio Franco 5f2f3f076d Add broken feature to old benchmarks (#1555)
Signed-off-by: Lucio Franco <[email protected]>
2019-09-13 14:04:23 -04:00
Taiki Endo efb27731ad timer: use our own AtomicU64 on targets with target_has_atomic less than 64 (#1538) 2019-09-13 10:18:32 -07:00
cynecx 578a9aec16 sync: replace deprecated mem::uninitialized usage with MaybeUninit (#1540) 2019-09-13 10:03:03 -07:00
Jonathan Bastien-Filiatrault 5b8fc19701 fs: propagate flush for stdout / stderr. (#1528) 2019-09-13 09:58:18 -07:00
Geoff Shannon c0a64d67ca chore: fix docs links (#1523) 2019-09-13 09:46:19 -07:00
Carl Lerche 6369d0f4f2 chore: add stability note to readme. (#1554) 2019-09-13 09:12:30 -07:00
Kirill Mironov f69ee652e6 tls: fix new temporary lifetime rustc error [E0597] (#1547)
Fixes: #1546
Signed-off-by: Kirill Mironov <[email protected]>
2019-09-11 10:22:04 -07:00
Jon Gjengset 9b3f8564af tls: Add get_ref and get_mut (#1537) 2019-09-04 17:45:53 -04:00
Ivan Petkov 9766cd644f process: omit several future types in favor of async/await (#1526) 2019-08-31 13:02:27 -07:00
Carl Lerche 431d4857e8 io: add Send / Sync impls for ReadHalf / WriteHalf (#1525) 2019-08-31 12:18:55 -07:00
Fenhl 26432355d5 tokio-process: Implement From<StdCommand> for Command (#1513) 2019-08-31 11:45:41 -07:00
Carl Lerche 2f91c85ad8 io: bring back split utility (#1521)
Bring back `split` utility as a free fn instead of a method on
`AsyncRead`. This utility wraps the `stream` in an `Arc` and uses mutual
exclusion to ensure correct access.

Additionally, the specialized `split_mut` fn on TcpStream and UdsStream
is promoted to `split`.
2019-08-30 20:46:07 -07:00
Fenhl 951827229a Add platform-specific methods to Command (#1516) 2019-08-30 18:28:41 -07:00
Geoff Shannon 383bb0a143 test: fix assert format args (#1520) 2019-08-30 14:03:37 -07:00
Benjamin Saunders d2bd6f5002 timer: Rename sleep to delay_for, reexport from tokio (#1518) 2019-08-30 10:23:54 -07:00
Carl Lerche 6a94d2cf4f tls: bump to v0.3.0-alpha.4 (#1515) 2019-08-30 10:20:44 -07:00
kellerkindt 4f99470d46 chore: fix compile error on latest nightly (#1512) 2019-08-30 09:15:05 -07:00
Jarred Nicholls 3d9134d13e executor: shut down idle threads in the blocking pool (#1514) 2019-08-30 08:26:16 -07:00
Sean McArthur 15dc0563b7 prepare v0.2.0-alpha.4 (#1509) 2019-08-29 12:59:10 -07:00
Sean McArthur 4e26258ac3 Re-add temporarily TcpStream::connect_std (#1508) 2019-08-29 11:45:17 -07:00
Carl Lerche a59e096c47 prepare v0.2.0-alpha.3 release (#1505) 2019-08-28 15:04:42 -07:00
Carl Lerche fc1640891e net: perform DNS lookup on connect / bind. (#1499)
A sealed `net::ToSocketAddrs` trait is added. This trait is not intended
to be used by users. Instead, it is an argument to `connect` and `bind`
functions.

The operating system's DNS lookup functionality is used. Blocking
operations are performed on a thread pool in order to avoid blocking the
runtime.
2019-08-28 13:25:50 -07:00
Jakub Beránek de9f05d4d3 docs: fix wording in tokio_process::Child documentation (#1502)
Fixes: #1494
2019-08-28 11:56:42 -04:00
Eliza Weisman 9c31797a08 net: switch from log to tracing (#1455)
* net: switch from `log` to `tracing`.

Motivation:

The `tracing` crate implements scoped, structured, context-aware
diagnostics, which can add significant debugging value over unstructured
log messages. `tracing` is part of the Tokio project. As part of the
`tokio` 0.2 changes, I thought it would be good to move over from `log`
to `tracing` in the tokio runtime.

Solution:

This branch replaces the use of `log` in `tokio-net` with
`tracing`. I've tried to leave all the instrumentation points more or
less the same, but modified to use structured fields instead of string
interpolation.

Notes:

I removed the timing in `Reactor::poll` in favor of simply adding a
`#[tracing::instrument]` attribute. Since the generated `tracing` span
will have enter and exit events, a `tracing::Subscriber`
implemementation can use those to record timestamps, and process that
timing data in a much more sophisticated manner than including it in a
log line.

We can add the timestamps back if they're desired.

Signed-off-by: Eliza Weisman <[email protected]>
2019-08-27 17:53:57 -07:00
Ömer Sinan Ağacan d1c58b7940 tokio: export RunError in tokio::runtime::current_thread (#1487)
This type is used in return type of `Runtime::run`, but because the type
was not exported it was opaque in the documentation of `Runtime`.
2019-08-27 12:26:48 -07:00
Jacob Pratt 5f74a99ea3 implement spawn_with_handle in tokio_executor (#1492)
This code directly relies on `future-preview`'s `RemoteHandle`, and
exposes it via a `spawn_with_handle` method that is identical to
`future-preview`'s implementation.

Related: #1180
2019-08-27 12:26:11 -07:00
Carl Lerche 08e20fcf6a fs: add support for non-threadpool executors (#1495)
Provides a thread pool dedicated to running blocking operations (#588)
and update `tokio-fs` to use this pool.

In an effort to make incremental progress, this is an initial step
towards a final solution. First, it provides a very basic pool
implementation with the intend that the pool will be
replaced before the final release. Second, it updates `tokio-fs` to
always use this blocking pool instead of conditionally using
`threadpool::blocking`. Issue #588 contains additional discussion around
potential improvements to the "blocking for all" strategy.

The implementation provided here builds on work started in #954 and
continued in #1045. The general idea is th same as #1045, but the PR
improves on some of the details:

* The number of explicit operations tracked by `File` is reduced only to
  the ones that could interact. All other ops are spawned on the
  blocking pool without being tracked by the `File` instance.

* The `seek` implementation is not backed by a trait and `poll_seek`
  function. This avoids the question of how to model non-blocking seeks
  on top of a blocking file. In this patch, `seek` is represented as an
  `async fn`. If the associated future is dropped before the caller
  observes the return value, we make no effort to define the state in
  which the file ends up.
2019-08-27 12:25:20 -07:00
Carl Lerche 08099bb2d3 net: rewrite TcpStream::connect with async fn (#1497)
This also removes `TcpStream::connect_std` as the conversion functions
from `std` need to be rethought. A note tracking this has been added
to #1209.
2019-08-27 12:08:28 -07:00
Newton Ni 807d536846 codec: fix infinite loop in tokio_codec::LinesCodec (#1489) 2019-08-26 13:38:52 -07:00
Danny Browning 654f9d703f tokio: expose signal feature (#1491)
Expose tokio_net::signal::ctrl_c via tokio::net::signal::ctrl_c as feature signal.
2019-08-22 09:12:00 -07:00
Jon Gjengset a285689664 net: shutdown TCP write when asked to shut down (#1488) 2019-08-21 10:41:51 -07:00
Gurwinder Singh 13930eff2a chore: two async_await feature remained (#1486) 2019-08-21 18:53:42 +09:00
Taiki Endo 24fb33e012 io: add AsyncReadExt::{chain, take} (#1484) 2019-08-20 20:09:07 -07:00
Taiki Endo a791f4a758 chore: bump to newer nightly (#1485) 2019-08-20 20:07:16 -07:00
Eliza Weisman 7e7a5147a3 executor: switch from log to tracing (#1454)
## Motivation

The `tracing` crate implements scoped, structured, context-aware
diagnostics, which can add significant debugging value over unstructured
log messages. `tracing` is part of the Tokio project. As part of the
`tokio` 0.2 changes, I thought it would be good to move over from `log`
to `tracing` in the tokio runtime. Updating the executor crate is an obvious
starting point. 

## Solution

This branch replaces the use of `log` in `tokio-executor` with
`tracing`. I've tried to leave all the instrumentation points more or
less the same, but modified to use structured fields instead of string
interpolation. I've also added a few `tracing` spans, primarily in
places where a variable is added to all the log messages in a scope.

## Notes

For users who are using the legacy `log` output, there is a feature flag
to enable `log` support in `tracing`. I thought about making this on by
default, but that would also enable the `tracing` dependency by default,
and it is only pulled in when the `threadpool` feature flag is enabled.
The `tokio` crate could enable the log feature in its default features
instead, since the threadpool feature is on by default in `tokio`. If
this isn't the right approach, I can change how `log` back-compatibility
is enabled.

We might want to consider adding more `tracing` spans in the threadpool
later. This could be useful for profiling, and for helping users debug
the way their applications interact with the executor. This branch is
just intended as a starting point so that we can begin emitting
`tracing` data from the executor; we should revisit what instrumentation
should be exposed, as well.

Signed-off-by: Eliza Weisman <[email protected]>
2019-08-20 12:44:26 -07:00
Jakub Beránek 2d56312b89 timer: introduce delay function shortcut (#1440)
This commit adds a simple delay shortcut to avoid writing Delay::new
everywhere and removes usages of Delay::new.
2019-08-20 08:39:55 -07:00
Ivan Petkov 357df38861 process: move into the tokio-net crate (#1475) 2019-08-19 19:42:54 -07:00
John-John Tedro 34a9dc2d76 Implement FusedStream and FusedFuture for Interval and Delay (#1476) 2019-08-19 10:21:34 -04:00
Ivan Petkov 68d5fcb8d1 docs: fix all rustdoc warnings (#1474) 2019-08-18 14:38:54 -07:00
Ivan Petkov 08b07afbd9 signal: remove new() constructors in favor of free functions (#1472)
* Also removed any `*_with_handle` related methods in favor of always
using the default reactor
2019-08-18 14:22:09 -07:00
Douman 7b0c60849c net: make default reactor guard public (#1468) 2019-08-18 11:36:21 -07:00
Ivan Petkov 6d8d388dc5 docs: add docs.rs metadata to build with all features (#1471) 2019-08-18 11:11:46 -07:00
Ivan Petkov bc61bd9d3d ci: ensure all tests are run for each feature (#1470)
* This includes running docs, examples, and lib tests for each added
feature, to ensure nothing is broken
2019-08-18 10:50:38 -07:00
Jakub Beránek a9585f0318 tokio-process: change CommandExt to a fully asynchronous Command struct (#1448)
Refs: #1371
2019-08-18 10:13:37 -07:00
Carl Lerche 88b4ec84d7 chore: prepare 0.2.0-alpha.2 release (#1465) 2019-08-17 23:34:25 -07:00
Philip Kannegaard Hayes 9f0daad5ac sync: fix fuzz_oneshot test by using instrumented loom::sync::Arc (#1464)
Since `tokio_sync::oneshot` makes a `CausalCell::with_mut()` mutable
access in the `Inner::drop()`, we must use the instrumented
`loom::sync::Arc`.

Uncovered by carllerche/loom#42
2019-08-17 21:30:31 -07:00
Carl Lerche c187cd75b6 signal: move into tokio-net (#1463) 2019-08-17 13:43:55 -07:00
Carl Lerche a83f5e4ba6 uds: move into tokio-net (#1462) 2019-08-16 14:42:05 -07:00
Carl Lerche 4935aae164 udp: remove files left over from moving tokio-udp (#1461) 2019-08-16 11:05:50 -07:00
Carl Lerche ba1829fd26 chore: rename ui-tests -> build-tests (#1460) 2019-08-16 09:26:56 -07:00
Carl Lerche ce7e60e396 udp: move tokio-udp into tokio-net (#1459) 2019-08-16 07:26:10 -07:00
Ivan Petkov d8b23ef852 signal: rename SignalKind methods (#1457)
This renames the SignalKind constructors to be a bit more readable
instead of using the signal names themselves
2019-08-15 21:09:09 -07:00
Carl Lerche 4788d3a9e3 tcp: move tokio-tcp into tokio-net (#1456) 2019-08-15 20:37:25 -07:00
Carl Lerche f1f61a3b15 net: reorganize crate in anticipation of #1264 (#1453)
Space is made to add `tcp`, `udp`, `uds`, ... modules.
2019-08-15 15:04:21 -07:00
Jakub Beránek d0a8e5d6f2 tokio-fs: rewrite std echo example using async/await (#1442)
This PR fixes the echo example in tokio-fs.

Refs: #1255
2019-08-15 13:10:17 -07:00
Carl Lerche 3b27dc31d2 threadpool: move threadpool into tokio-executor (#1452)
The threadpool is behind a feature flag.

Refs: #1264
2019-08-15 13:09:02 -07:00
Douman 37131b2114 runtime: refactor thread-local setters (#1449) 2019-08-15 13:00:57 -07:00
Carl Lerche 8538c25170 reactor: rename tokio-reactor -> tokio-net (#1450)
* reactor: rename tokio-reactor -> tokio-net

This is in preparation for #1264
2019-08-15 11:04:58 -07:00
Jakub Beránek 7b6438a172 tokio: rewrite print_each_packet example using async/await (#1446)
This PR fixes the print each packet example in tokio.

Refs: #1201
2019-08-15 10:42:34 -07:00
Carl Lerche 9de7083be8 executor: move current-thread into crate (#1447)
The `CurrentThread` executor is exposed using a feature flag.

Refs: #1264
2019-08-15 09:52:25 -07:00
John Doneth 8d55f98f6f udp: update tokio_udp::UdpFramed to std::future (#1370) 2019-08-14 11:18:21 -07:00
Taiki Endo 999a600494 io: add async BufReader/BufWriter (#1438) 2019-08-14 10:24:07 -07:00
Ilya Lakhin fb9809c068 executor, threadpool: forward port fix from #1155 (#1433)
Add executor::exit, allowing other executors inside threadpool::blocking.
2019-08-13 21:12:49 -07:00
Douman 517162792f macros: upgrade syn/quote (#1432) 2019-08-13 21:11:26 -07:00
Geoff Shannon fe90d61446 test: add a block_on function to tokio-test (#1431) 2019-08-13 21:10:26 -07:00
Ivan Petkov 338b37884a signal: Add SignalKind for registering signals more easily (#1430)
This avoids having consumers import libc for common signals, and it
improves discoverability since users need not be aware that libc
contains all supported constants.
2019-08-13 21:07:22 -07:00
Ivan Petkov 513326e01d signal: remove driver task for Windows event implementation (#1429)
Windows guarantees handler routines are always invoked in a new thread
(https://docs.microsoft.com/en-us/windows/console/handlerroutine), so we
don't need to use the handler-wake-another-driver technique used in the
Unix implementation

By broadcasting the event notifications from the handler, we no longer
need the Driver task to be spawned, which fixes the starvation issue if
the executor which runs the Driver task goes away

Also changed the behavior so that the default event handler runs if
all listeners for CTRL_{C, BREAK} events go away.
2019-08-13 21:01:06 -07:00
Ivan Petkov 73a91ad7b3 signal: delete blocking Read/Write impls on ChildStd{in, out, err} (#1428) 2019-08-13 20:53:02 -07:00
Taiki Endo 930cce8677 chore: update futures-preview to 0.3.0-alpha.18 (#1427) 2019-08-10 14:09:28 -07:00
Taiki Endo 6a125082e4 chore: apply unreachable_pub and missing_debug_implementations to all crates (#1424) 2019-08-11 04:28:52 +09:00
Taiki Endo d9f9c5658f chore: bump to newer nightly (#1426) 2019-08-11 02:01:20 +09:00
Taiki Endo fff39c03b1 ci: deny warnings in cirrus (#1425) 2019-08-11 01:41:51 +09:00
Tomasz Miąsko 756606a58b uds: implement split and split_mut for UnixStream (#1395)
This mirrors split API available in TcpStream.
2019-08-09 12:50:18 -07:00
Ran Benita e3b4c99a33 codec: a few suggestions (#1418)
How the buffer is managed is often critical for performance. Not
taking care of it will be catastrophic for performance beyond the
initial buffer size with the current implementation (a loop of
`reserve(1)`).
2019-08-09 12:20:31 -07:00
Taiki Endo 42fa0c28d3 timer: use std::sync::atomic::AtomicU64 instead of own AtomicU64 (#1421) 2019-08-10 03:42:03 +09:00
Taiki Endo f7b41c9dcc macros: improve error messages (#1420) 2019-08-09 10:28:22 -07:00
Taiki Endo 73102760cf chore: change default lint level to warning and deny warnings in CI (#1416) 2019-08-10 00:07:57 +09:00
Douman 18833a8e67 macros: Error on function with arguments (#1419) 2019-08-09 11:04:41 -04:00
tmiasko eba8bf2b4b io: implement AsyncWrite for Vec<u8> (#1409) 2019-08-08 20:55:27 -07:00
David Kellum 790d649dc5 update (dev dep) env_logger to latest 0.6 (#1390) 2019-08-08 20:37:32 -07:00
Lucio Franco 50e5d401df chore: prepare for v0.2.0-alpha.1 release (#1410) 2019-08-08 12:48:53 -07:00
Carl Lerche 2e69f2a7fd sync: track upstream loom changes (#1407) 2019-08-07 23:24:22 -07:00
Carl Lerche 962521f449 chore: enable full CI run (#1399)
* update all tests
* fix doc examples
* misc API tweaks
2019-08-07 20:02:13 -07:00
Carl Lerche 831be9c08e executor: remove unused dependency (#1406) 2019-08-07 19:55:42 -07:00
Carl Lerche 23c380a78f sync: track loom changes (#1405) 2019-08-07 15:38:34 -07:00
Lucio Franco 0a05332648 Remove git dep and add macro examples (#1404)
Signed-off-by: Lucio Franco <[email protected]>
2019-08-07 15:02:38 -07:00
Lucio Franco 7268b0bb3a Migrate threadpool to futures-util (#1403)
* Migrate threadpool to futures-util

Signed-off-by: Lucio Franco <[email protected]>

* fmt
2019-08-07 16:26:44 -04:00
Lucio Franco 6412389bba executor: update park implementation (#1402)
Signed-off-by: Lucio Franco <[email protected]>
2019-08-07 13:01:13 -07:00
tmiasko 53a94c025d io: implement AsyncBufRead for &[u8] and Cursor (#1397)
* `impl AsyncRead for &[u8]`
* `impl AsyncBufRead for &[u8]`
* `impl<T: AsRef<[u8]> + Unpin> AsyncRead for Cursor<T>`
* `impl<T: AsRef<[u8]> + Unpin> AsyncBufRead for Cursor<T>`
2019-08-07 12:57:37 -07:00
Gurwinder Singh 7174c63bf9 codec: move length delimited codec to tokio-codec (#1401) 2019-08-07 12:10:05 -07:00
Ivan Petkov cb2336ff3d process: Misc polish (#1400)
* Denied all warnings in tests, and denied rust_2018_idioms violations
* Bumped the crate version and set publish = false
* Pruned dependencies:
 - Only pull in tokio-sync on windows where it is used
 - Removed unused dev-dependencies
* Switch to Async{Read, Write} traits from tokio-io rather than
futures-io
* Use #[tokio::test] where possible
* Removed deprecated items
* Fix all doc examples
2019-08-07 10:38:45 -07:00
Carl Lerche 47e2ff48d9 tokio: fix API doc examples (#1396) 2019-08-06 14:03:49 -07:00
Carl Lerche 2f43b0a023 sync: polish and update API doc examples (#1398)
- Remove `poll_*` fns from some of the sync types.
- Move `AtomicWaker` and `Lock` to the root of the `sync` crate.
2019-08-06 13:54:56 -07:00
Carl Lerche 05d00aebb7 uds: remove poll_* fns in favor of async fns (#1394) 2019-08-05 15:05:02 -07:00
Carl Lerche 62733a6594 udp: remove poll_* fns in favor of async fns (#1393)
This removes the need for manual futures.
2019-08-05 14:18:18 -07:00
Carl Lerche 6d8cc4e475 tcp: update API documentation (#1392) 2019-08-05 11:50:55 -07:00
Carl Lerche 6cbe3d4f82 fs: use async fn instead of custom futures (#1381)
Also update all the doc examples.
2019-08-04 11:24:30 -07:00
Carl Lerche 337646b97f tokio: re-export future/stream utils (#1387) 2019-08-03 21:08:29 -07:00
Taiki Endo 0bb015588a codec: add AsyncBufRead/BufRead implementations (#1385)
* AsyncBufRead for FramedWrite2<T>
* BufRead for FramedWrite2<T>
* AsyncBufRead for Fuse<T, U>
* BufRead for Fuse<T, U>
2019-08-03 20:15:50 -07:00
Steven Fackler 63377e2110 Add AsyncWriteExt::shutdown (#1382) 2019-08-03 00:51:24 -04:00
Carl Lerche 878503f965 docs: update API documentation for some crates (#1380)
Updates API documentation for

- tokio-buf
- tokio-codec
- tokio-current-thread
- tokio-executor
2019-08-02 14:35:32 -07:00
Carl Lerche 2c01b3e0e0 io: remove util from default features (#1379)
Sub-crates should require opting into features.
2019-08-02 12:59:24 -07:00
Carl Lerche ee9105d166 tokio: add async io traits to prelude (#1378) 2019-08-02 12:50:40 -07:00
Lucio Franco 5a4f849bba tokio: update tinyhttp example to async/await (#1372) 2019-08-02 12:24:15 -07:00
Lucio Franco ff41108834 io: move io helpers back into tokio-io (#1377)
Utilities are made optional with a feature flag.
2019-08-02 12:23:44 -07:00
Lucio Franco 6b202722ea io: Add AsyncWriteExt::flush (#1376)
* io: Add `AsyncWriteExt::flush`

* fmt

* fix clippy
2019-08-02 13:53:49 -04:00
Lucio Franco 144d980e5c tokio: update connect to async/await (#1375) 2019-08-02 10:03:06 -07:00
Lucio Franco 81d789b88f tokio: Update proxy to async/await (#1373) 2019-08-01 20:03:34 -07:00
Lucio Franco 634c19582f chore: add rust-toolchain file to track nightly version (#1374) 2019-08-01 20:00:55 -07:00
Ivan Petkov ff922bbe6d signal: Change constructors to return a result instead of lazy future (#1340) 2019-07-30 18:23:26 -07:00
Gurwinder Singh bf38631d6a chore: Fix spelling mistake (#1359) 2019-07-30 10:53:11 -07:00
Taiki Endo 6dda866191 tokio: re-enable StreamExt (#1362) 2019-07-30 09:55:34 -07:00
Taiki Endo 03e450deb1 sync: switch branch of loom dev-dependency to master (#1367)
* sync: switch branch of loom dev-dependency to master

* replace loom::fuzz with loom::model
2019-07-30 10:11:46 -04:00
andy finch fbf90e6356 Update process to use std::future (#1343) 2019-07-29 18:36:11 -07:00
Shell Chen 74168ae82f tcp: add async fn TcpStream::peek (#1360)
* tcp: add `async fn TcpStream::peek`

* tcp: apply rustfmt on tests
2019-07-26 10:55:02 -04:00
John Doneth d038009e7d Update chat example to async/await (#1349) 2019-07-25 19:44:23 -04:00
John Doneth 132e9f1da5 Update examples to return Result (#1305)
* update echo-udp

* update echo

* update hello_world

* update udp-client

* rustfmt

* remove send & sync

* rebase & change new updated examples
2019-07-25 16:47:31 -04:00
Taiki Endo fe021e6c00 ci: enable clippy lints (#1335) 2019-07-26 03:47:14 +09:00
Lucio Franco f311ac3d4f buf: Inital pass at updating BufStream (#1355) 2019-07-25 14:21:48 -04:00
Shell Chen 298be80249 tokio: include async-trait feature for uds (#1352) 2019-07-25 08:07:33 -07:00
John Doneth 79b017c773 Export LinesCodecError (#1350) 2019-07-24 15:26:41 -04:00
Taiki Endo ca0e5cc670 add TryFrom/From implementations (#1347)
* TryFrom<net::TcpListener> for TcpListener
* TryFrom<net::TcpStream> for TcpStream
* TryFrom<net::UdpSocket> for UdpSocket
* TryFrom<net::UnixDatagram> for UnixDatagram
* TryFrom<net::UnixListener> for UnixListener
* TryFrom<net::UnixStream> for UnixStream
* TryFrom<UnixDatagram> for mio_uds::UnixDatagram
* TryFrom<File> for io::File
* From<io::File> for File
2019-07-24 09:26:12 -07:00
Douman 59bc364a0e macros: detect double test attribute (#1336) 2019-07-22 09:28:07 -07:00
Taiki Endo e88d10a3cb chore: bump to newer nightly (#1338) 2019-07-22 06:04:02 +09:00
Ivan Petkov a3b8d82711 Merge tokio-process into tokio
Original repo can be found at https://github.com/alexcrichton/tokio-process/
2019-07-21 11:08:16 -07:00
Ivan Petkov d9688bc094 signal: change unix::Signal to return () instead of signum (#1330)
* This simplifies the API surface by returning () instead of the signal
number that was used during registration. This also more closely mirrors
the cross-platform `CtrlC` event stream API
* This is a **breaking change**
2019-07-20 15:12:53 -07:00
Ivan Petkov 320a5fdca7 signal: replace windows::Event with windows::CtrlBreak (#1331)
* Add a new `windows::CtrlBreak` struct which wil represent a stream of
CTRL_BREAK_EVENT signals on Windows systems
* The `windows::Event` type is no longer publicly accessible and is
replaced by using `CtrlC` or `windows::CtrlBreak`.

[breaking-change]
2019-07-20 10:50:27 -07:00
Taiki Endo 9af07ce208 chore: remove redundant field names in struct literals (#1334) 2019-07-20 10:43:19 -07:00
Taiki Endo 1b2d997863 chore: use ptr::{null, null_mut} instead of 0 as *{const, mut} (#1333) 2019-07-20 10:41:02 -07:00
Taiki Endo 7a52ddcd09 chore: remove unnecessary conversion (#1332) 2019-07-20 12:10:42 -04:00
Carl Lerche 9d3e5aac08 tokio: remove Send + 'static requirement from block_on (#1329)
Removes the `Send` requirement to futures passed to `Runtime::block_on`.
Previously, `block_on` was implemented by sending the future to a
runtime thread. In order to do this, the future must be Send.

The reason why the future is sent to the pool is because we cannot
guarantee, while off the pool, that a reactor / timer thread is running.
This is due to a limitation in the current version of tokio-threadpool.
There is a plan to fix this (#1177), but the proper fix is non trivial.

In order to unblock APIs that require this, this patch updates the
runtime to spawn an always running thread containing a reactor and
timer. All calls to `block_on` will use that reactor and timer.
2019-07-19 17:25:04 -07:00
Carl Lerche a99fa6e096 chore: remove tokio-futures facade crate (#1327)
This switches from using the tokio-futures facade to referencing
futures-* crates directly.
2019-07-19 13:11:46 -07:00
David Kellum b89ed00a0d Remove last non-dev dependency on rand crate (#1324)
Use std RandomState for XorShift seeding. This allows dropping _rand_
crate dep here, accept as a dev dependency for tests or benchmarks.
2019-07-19 12:12:32 -07:00
Dylan Frankland 12ce75f088 fs: add remove_dir_all and RemoveDirAllFuture (#1325)
Adds the sister function to `remove_dir` and mirrors the `create_dir_all` that's already exposed.
2019-07-19 12:09:53 -07:00
Taiki Endo a88308ed9f tokio: add AsyncReadExt::read_to_string (#1326) 2019-07-19 11:50:00 -07:00
João Oliveira a298472da8 tokio-tls: enable Send and Sync (#1317)
-  update 0 as *mut () calls to std::ptr::null_mut()
- impl Send and Sync for AllowStd
2019-07-19 10:21:26 -07:00
Shell Chen d0bb16192b timer: change Into to From trait for Elapsed (#1322) 2019-07-17 09:04:59 -04:00
Shell Chen a18ddb3b61 timer: impl Into<std::io::Error> for Elpased (#1321)
That convert Elpased to ErrorKind::TimedOut
2019-07-16 20:22:03 -07:00
Jon Gjengset 003b4d8074 Get rid of Enter for with_default (#1315)
We want executors to enforce that there are never multiple active at the
same time. This is ensured through `Enter`, which will panic if you
attempt to create more than one. However, by requiring you to pass an
`&mut Enter` to `executor::with_default`, we were *also* disallowing
temporarily overriding the current executor.

This patch removes that requirement.
2019-07-16 14:29:35 -04:00
Yin Guanhao 6d186fe40e Replace (some) uninitialized with MaybeUninit (#1295) 2019-07-16 10:47:46 -07:00
Diggory Blake 0d99ddd4f4 tcp: implement "split_mut" for TcpStream (#1289) 2019-07-16 10:28:00 -07:00
João Oliveira 448d9d2eab tls: update to std-future (#1224) 2019-07-16 10:26:08 -07:00
David Kellum 0de3a69eb4 fs: drop deprecated tempdir crate use in tests (#1312)
In particular because it pulls in old rand duplicates. Replace use
with tempfile::tempdir() which has been available since tempfile
3.0.0.
2019-07-15 15:32:33 -07:00
John Doneth 61aee5fc28 examples: pdate tinydb example (#1288)
Update tinydb example to use async / await.
2019-07-15 15:19:36 -07:00
Sean McArthur 7f7f74985e io: Minor adjustments to tokio-test IO (#1306)
This also re-exports `bytes::{Buf, BufMut}` from `tokio-io`.
2019-07-15 14:53:16 -07:00
Jon Gjengset e6cf976662 tokio: include async-traits feature (#1314)
The `tokio` facade crate will depend on the `async-traits` feature flag in
sub crates.
2019-07-15 14:02:14 -07:00
Taiki Endo b14e189e44 add #[must_use] to more futures and streams (#1309) 2019-07-15 13:28:56 -07:00
Taiki Endo 2dde2b448f Fix import of ready macro 2019-07-15 11:52:13 -07:00
Taiki Endo 6742816e78 tokio: add AsyncBufReadExt::lines 2019-07-15 11:52:13 -07:00
Taiki Endo ab040bb498 tokio: add AsyncBufReadExt::read_line 2019-07-15 11:52:13 -07:00
Taiki Endo 0cfa120ba8 tokio: add AsyncBufReadExt::read_until 2019-07-15 11:52:13 -07:00
Taiki Endo 5774a9cd64 io: add AsyncBufRead trait 2019-07-15 11:52:13 -07:00
John Doneth da49ede41e update udp-codec example (#1293) 2019-07-15 14:14:03 -04:00
Carl Lerche d224d6415e chore: indicate the master branch docs are old. (#1304)
Fixes #1292
2019-07-15 10:44:47 -07:00
Gurwinder Singh 83273b8b50 chore: use ready macro from futures-core (#1300) 2019-07-15 10:43:54 -07:00
Taiki Endo ca708d6d87 chore: update rand dependency to 0.7 (#1302) 2019-07-15 10:13:10 -07:00
matthieugras 0b75c0c53d executor: block thread when needed in block fn (#1303)
Fix #1296
2019-07-15 08:56:20 -07:00
Alex Gaynor 5fbb36a060 reactor: bump parking_lot dependency (#1298) 2019-07-14 09:28:54 -07:00
Gurwinder Singh c897a5b696 Re-export tokio-fs (#1287) 2019-07-14 12:03:49 -04:00
Sean McArthur 48d7f7b931 tokio-test: add tokio_test::io mock builder 2019-07-12 11:19:12 -07:00
Carl Lerche 2291823181 tokio: re-export correct tokio-uds version (#1286)
An earlier PR (#1282) re-exported the version from crates.io and not git
master.
2019-07-11 10:27:51 -07:00
andy finch 795e02f4c6 fs: update to use std::future (#1269) 2019-07-11 09:05:49 -07:00
Carl Lerche 7ac8bfc821 chore: bump to newer nightly (#1284) 2019-07-10 14:36:36 -07:00
Carl Lerche a79483750f tokio: update echo example (#1283) 2019-07-10 14:21:20 -07:00
Carl Lerche 3855f373d3 tokio: re-export tokio-uds (#1282)
The tokio-uds crate has been previously updated to std::future. This
commit enables the re-export in the tokio facade crate.
2019-07-10 11:21:27 -07:00
Carl Lerche bd3f3270db tokio: update threaded runtime to std::future (#1280)
re-enables the threaded runtime and sets it (again) as the default.
2019-07-10 11:21:06 -07:00
Taiki Endo e5525628cd chore: remove usage of deprecated ONCE_INIT (#1281) 2019-07-10 08:35:07 -07:00
Carl Lerche f1b8a318d9 tokio: add AsyncReadExt::read_to_end (#1279) 2019-07-09 16:17:58 -07:00
Carl Lerche 64343f1b78 tokio: add AsyncWriteExt::write_all (#1277) 2019-07-09 12:37:14 -07:00
Ruben De Smet 82795184c1 tokio: rewrite examples with async. (#1228) 2019-07-09 11:21:12 -07:00
Thomas Lacroix f529928d87 chore: script updating versions in links to docs.rs (#1249) 2019-07-09 11:19:38 -07:00
Ivan Petkov 461eebe612 signal: Replace ctrl_c with a CtrlC struct (#1273)
* Add a new `CtrlC` struct which will represent a stream of SIGINT
signals on Unix or the CTRL_C event on Windows
* `CtrlC` implements `Stream<Output = ()>` rather than `IoSteam` as
previously
2019-07-09 08:48:46 -07:00
Gurwinder Singh 407d15cf93 chore: Add link to docs (#1276) 2019-07-09 11:21:29 -04:00
Yin Guanhao 80915906d8 uds: update to std-future (#1227) 2019-07-08 14:58:40 -07:00
Yin Guanhao 88e775dcf0 udp: UdpSocket split support (#1226) 2019-07-08 14:47:31 -07:00
Carl Lerche 8b49a1e05f chore: update examples link in README (#1274) 2019-07-08 13:34:39 -07:00
Carl Lerche 8fa1510d67 timer: fix build (#1275) 2019-07-08 11:23:15 -07:00
Reto Kaiser 7797a377c3 current-thread: make tokio_current_thread::Handle Sync (#1119) 2019-07-08 10:25:34 -07:00
Steven Fackler b62d224fac timer: fix Handle::timeout (#1093)
The old implementation didn't work for Timeout<Stream>, since the method
took a deadline rather than a timeout.
2019-07-08 10:18:37 -07:00
Aaron Hill d4803bc868 Use Sink trait from futures-sink-preview (#1244) 2019-07-08 09:56:11 -07:00
Thomas Lacroix e07a03b3c5 signal: update instructions in Ctrl-C example (#1270)
Fixes: #1248
2019-07-07 09:49:19 -07:00
Taiki Endo 7b86acb71d chore: Update futures-preview to 0.3.0-alpha.17 (#1267) 2019-07-04 14:34:57 -07:00
Steffen Butzer 0651f09427 Remove usage of deprecated std::error::Error methods (#1206) (#1245) 2019-07-03 23:06:03 -07:00
Thomas Lacroix 516251052d Add missing links in README.md (#1233)
Fixes: #1229
2019-07-03 22:59:10 -07:00
Ivan Petkov cbad83f362 signal: migrate to std::futures (#1218)
Migrate to std::futures and the futures 0.3 preview and use async/await
where possible

**Breaking change:** the IoFuture and IoStream definitions used to refer
to Box<dyn Future> and Box<dyn Stream>, but now they are defined as
Pin<...> versions which are technically breaking.

No other breaking or functional changes have been made
2019-07-03 10:40:59 -07:00
Eliza Weisman bd9760e124 add release documentation to CONTRIBUTING.md (#1171)
## Motivation

Currently, the process for releasing a new version of a Tokio crate is
somewhat complex, and is not well-documented. To make it easier for
contributors to release minor versions more frequently, there should be
documentation describing this process.

## Solution

This branch adds a section to `CONTRIBUTING.md` describing how to
release a new version of a Tokio crate. The steps are based on those
described by @carllerche in an offline conversation.

I've also added a quick shell script to actually publish new crate 
versions. This should make it harder to make mistakes when 
publishing.

Signed-off-by: Eliza Weisman <[email protected]>
2019-07-03 10:18:02 -07:00
Carl Lerche 3e898f58a5 tcp: add ascyc fn TcpListener::accept (#1242)
Refs: #1209
2019-07-03 09:49:56 -07:00
Ivan Petkov c531865d2c ci: don't generate docs for deps on FreeBSD (#1241) 2019-07-03 09:41:35 -07:00
Ivan Petkov 722eb257be ci: scope each tests/examples invocation to a specific crate (#1238) 2019-07-03 08:49:05 -07:00
Taiki Endo ceed29586b io: fix documents (#1231) 2019-07-01 20:44:12 -07:00
Carl Lerche 70eca184f0 tokio: re-enable timer in runtimes (#1237)
This also brings back the timer tests in the tokio crate.
2019-07-01 18:27:13 -07:00
Carl Lerche b2c777846e timer: finish updating timer (#1222)
* timer: restructure feature flags
* update timer tests
* Add `async-traits` to CI

This also disables a buggy `threadpool` test. This test should be fixed in the future.

Refs #1225
2019-06-30 08:48:53 -07:00
Lucio Franco 8e7d8af588 docs: add note in the readme about the master branch (#1230) 2019-06-29 21:47:20 -04:00
Yin Guanhao 7380dd2482 TcpSocket specialized split (#1217) 2019-06-28 23:36:49 -07:00
Eliza Weisman af46eac583 chore: remove tokio-trace, add "Related Projects" to README (#1221)
## Motivation

The `tokio-trace` and `tokio-trace-core` crates have been renamed to
`tracing` and `tracing-core`, and moved to their own repository
(`tokio-rs/tracing`).

## Solution

This branch removes `tokio-trace` and `tokio-trace-core` from the
`tokio` repository. In addition, I've added a "Related Projects" section
to the root README, which lists `tracing` (as well as  `mio`, and
`bytes`) as other libraries maintained by the Tokio project. I thought
that this would help folks looking for `tokio-trace` here find it in its
new home.

In addition, it changes `tokio` to depend on `tracing-core` rather than
`tokio-trace-core`.

Closes #1159

Signed-off-by: Eliza Weisman <[email protected]>
2019-06-28 13:13:46 -07:00
Carl Lerche e7488d983e threadpool: update to std::future (#1219)
An initial pass at updating `tokio-threadpool` to `std::future`. The
codebase and tests both now run using `std::future` but the wake
mechanism is not ideal. Follow up work will be required to improve on
this.

Refs: #1200
2019-06-27 22:30:56 -07:00
Sean McArthur e4415d986a sync: change oneshot poll_close to poll_closed
The action of `Sender::poll_close` is to check if the receiver has been
closed, not to try to close the sender itself. So change to
`poll_closed`.
2019-06-27 13:56:58 -07:00
Carl Lerche ff906acdfb ci: disable cache on cirrus (#1215)
Caching takes longer than rebuilding
2019-06-27 12:08:43 -07:00
Carl Lerche 32ceccb465 sync: add async APIs to oneshot and mpsc (#1211)
Adds:

- oneshot::Sender::close
- mpsc::Receiver::recv
- mpsc::Sender::send

Also renames `poll_next` to `poll_recv`.

Refs: #1210
2019-06-27 11:33:36 -07:00
Douman 0af05e7408 macros: allow configuring runtime used by main macro (#1185) 2019-06-27 10:40:21 -07:00
jesskfullwood 6b9e7bdace codec: update to use std-future (#1214)
Strategy was to

- copy the old codec code that was temporarily being stashed in `tokio-io`
- modify all the type signatures to use Pin, as literal a translation as possible
- fix up the tests likewise

This is intended just to get things compiling and passing tests. Beyond that there is surely
lots of refactoring that can be done to make things more idiomatic. The docs are unchanged.

Closes #1189
2019-06-27 10:10:29 -07:00
Carl Lerche ed4d4a5353 chore: format code and enable rustfmt CI task (#1212) 2019-06-27 00:05:01 -07:00
Carl Lerche 1f47ed3dcc tokio: rewrite io_read.rs test to use async/await (#1207)
This simplifies the test
2019-06-26 17:06:56 -07:00
Carl Lerche e9aaacddbd tokio: re-export sync::{lock,mpsc} (#1208)
These types have been updated already.
2019-06-26 16:54:15 -07:00
Carl Lerche 11f6b2862f tokio: move I/O helpers to ext traits (#1204)
Refs: #1203
2019-06-26 14:42:19 -07:00
Carl Lerche 8404f796ac test: get cargo test --tests working (#1205)
Broken tests are disabled
2019-06-26 14:40:52 -07:00
Yin Guanhao 6316aa1d0b Update tokio-udp to use std-future (#1199) 2019-06-26 14:41:36 -04:00
Bhargav 0784dc2767 tokio: add read_exact method (#1202) 2019-06-26 11:36:09 -07:00
Denis dd126c2333 Implement TryFrom to transform various I/O primitives into their mio counterparts (#1158)
* `TryFrom<TcpListener> for mio::net::TcpListener`
* `TryFrom<TcpStream> for mio::net::TcpStream`
* `TryFrom<UdpSocket> for mio::net::UdpSocket`
* `TryFrom<UnixListener> for mio_uds::UnixListener`
* `TryFrom<UnixStream> for mio_uds::UnixStream`
2019-06-26 08:51:38 -07:00
Lucio Franco 3cc33dca7c sync: Fix lock test to actually test the inner lock value (#1197)
* sync: Fix lock test to actually test the returned value

* Update lock test to use task.poll
2019-06-26 11:32:41 -04:00
Carl Lerche dc5fa80a09 macros: re-export main macro from tokio (#1198)
Includes minor fixes and a very basic example.

Fixes #1183
2019-06-25 20:14:21 -07:00
Zahari Dichev 455782b964 trace: Allow setting event parents explicitly (#1109)
## Motivation 

As mentioned in tokio-rs/tracing#1100  it makes sense to be able to set
the parents of events explicitly.

## Solution 

For that to happen the Parent type is extracted from span.rs and a
`parent` field is added to Event. Additionally the appropriate macros
arms are added with corresponding tests as described in
tokio-rs/tracing#1100

Closes tokio-rs/tracing#1100

Signed-off-by: Zahari Dichev <[email protected]>
2019-06-25 15:12:52 -07:00
Lucio Franco 29e417c257 tokio: Add io copy, read, and write (#1187) 2019-06-25 16:51:49 -04:00
Ivan Petkov 9df1140340 signal: factor out event delivery into its own module to share between Unix and Windows (#1174)
Today the Unix and Windows implementations have similar yet differing
implementations of hooking into OS events and propagating them to any
listening futures. Rather than re-implement the same behavior two
different ways, we should factor out any commonality into a shared
module and keep the Unix/Windows modules focused solely on OS
integrations.

Reusing the same implementation across OS versions also allows for more
consistent behavior between platforms, which also makes squashing bugs
much easier.

This change introduces the `registry` module which handles creating and
initializing a global map of signals/events and their registered
listeners. Each OS specific module is expected to implement the OS hooks
which delegate to invoking the registry module's methods for
distributing the event notifications.

# Use registry module for Windows implementation

Note this still uses the same architecture as previously: a driver task
is spawned by the first registered event, and that task is responsible
for delivering any events to registered futures. (If that first event
loop goes away, all events will deadlock). A solution to this issue will
be explored at a later time.
2019-06-25 13:07:59 -07:00
Lucio Franco e2b4bdb647 sync: Add LockFuture for Lock (#1184) 2019-06-25 10:42:35 -07:00
Ivan Petkov c6defbce4b process: Move files to their own directory 2019-06-24 17:31:47 -07:00
Ivan Petkov b7846a4e2f process: Remove unneeded files 2019-06-24 17:31:00 -07:00
Ivan Petkov cb8607a816 process: Update to 2018 edition 2019-06-24 17:29:33 -07:00
Ivan Petkov 27c15471c1 process: Run cargo fmt 2019-06-24 17:29:33 -07:00
Ivan Petkov 0ab25878bd process: Update README 2019-06-24 17:29:32 -07:00
Eliza Weisman 448302c3d4 trace: Improve documentation (#1148) 2019-06-24 19:22:05 -05:00
Ivan Petkov 934a1467d4 process: Update CHANGELOG 2019-06-24 17:12:17 -07:00
Ivan Petkov 4d639e246b process: Update Cargo.toml 2019-06-24 17:10:58 -07:00
Ivan Petkov ff5381de8d process: Update license files 2019-06-24 17:10:58 -07:00
Ivan Petkov 061452dc01 process: Delete flaky and (now) unused test 2019-06-24 17:10:58 -07:00
Ivan Petkov a6b2682309 process: Bump to 0.2.4 2019-06-24 16:57:20 -07:00
Ivan Petkov cf84a59e5a process: Don't kill child on drop if already successfully killed 2019-06-24 16:57:20 -07:00
Ivan Petkov e90e33d5df process: Add unit tests for dropping killing dropped children 2019-06-24 16:57:20 -07:00
Ivan Petkov fa5da27d98 process: Utilize a global orphan process queue to avoid leaks 2019-06-24 16:57:20 -07:00
Ivan Petkov ecaa069f0f process: Implement a queue for repeatedly attempting to reap orphaned processes 2019-06-24 16:57:20 -07:00
Ivan Petkov fc15d7d4a4 process: Only pull in mio dependency on unix platforms 2019-06-24 16:57:20 -07:00
Ivan Petkov a70a3b599a process: ci: move cargo tool installation to after_success 2019-06-24 16:57:20 -07:00
Ivan Petkov 26faefcc34 process: ci: enable clippy checks as part of the build 2019-06-24 16:57:19 -07:00
Ivan Petkov f16725ea9f process: Fix clippy warnings 2019-06-24 16:57:19 -07:00
Ivan Petkov caf43221b5 process: ci: fix cargo binary caching 2019-06-24 16:57:19 -07:00
Ivan Petkov 93680357dd process: Fix drop_kills test when running on macOS with a single thread 2019-06-24 16:57:19 -07:00
Ivan Petkov 784d21ae31 process: Try pinning mio to 0.1.16 2019-06-24 16:57:19 -07:00
Ivan Petkov 0938ccfefd process: ci: cache cargo tarpaulin build 2019-06-24 16:57:19 -07:00
Ivan Petkov 6fa2fdab44 process: Ensure all tests are run with an explicit timeout 2019-06-24 16:57:19 -07:00
Ivan Petkov d0d13d0bd0 process: Change codecov comment behavior to default 2019-06-24 16:57:19 -07:00
Ivan Petkov 8a1777b800 process: Rename EventedReaper to Reaper 2019-06-24 16:57:18 -07:00
Ivan Petkov 42d0f53ddb process: Optimize out the "reaped" flag 2019-06-24 16:57:18 -07:00
Ivan Petkov db0c4147c8 process: Refactor Unix process handling 2019-06-24 16:57:18 -07:00
Ivan Petkov 10fd2afd18 process: Simplify child IO registration 2019-06-24 16:57:18 -07:00
Ivan Petkov 83a55601ef process: Move src/unix.rs to src/unix/mod.rs 2019-06-24 16:57:18 -07:00
Ivan Petkov b37120f61c process: Update line-by-line doc example to be more flexible 2019-06-24 16:57:18 -07:00
Ivan Petkov 91dbf24cf4 process: Update min supported rust version as per the Tokio project policy 2019-06-24 16:57:18 -07:00
Ivan Petkov c78fd6d6c5 process: Update Travis link from .org to .com 2019-06-24 16:57:18 -07:00
Ivan Petkov 025474dfbb process: ci: Install cargo-tarpaulin *after* initial tests 2019-06-24 16:57:18 -07:00
Ivan Petkov e7dfcf90fe process: ci: Enable code coverage tracking via codecov.io 2019-06-24 16:57:17 -07:00
Ivan Petkov ecdfe4c474 process: ci: collect code coverage info via cargo-tarpaulin 2019-06-24 16:57:17 -07:00
Ivan Petkov 37b4efb9e2 process: Bump version to 0.2.3 2019-06-24 16:57:17 -07:00
Ivan Petkov c94f607f1b process: Fix some test case deprecation warnings 2019-06-24 16:57:17 -07:00
Ivan Petkov 76438c9e70 process: Implement AsRawHandle for ChildStd{in, out, err} for parity 2019-06-24 16:57:17 -07:00
Yuya Nishihara e0e9594f71 process: Implement AsRawFd for ChildStd* structs 2019-06-24 16:57:17 -07:00
Yuya Nishihara 3b43262a10 process: Implement AsRawFd for inner Fd<T> wrappers and use it instead of self.0 2019-06-24 16:57:17 -07:00
Ivan Petkov 5f18bf669f process: Bump minimum supported rustc version to 1.26 2019-06-24 16:57:17 -07:00
Ivan Petkov 1581c8b475 process: Bump minimum required version of tokio-signal to 0.2.5 2019-06-24 16:57:16 -07:00
Ivan Petkov d3b2efc815 process: Add regression test for signal starvation 2019-06-24 16:56:53 -07:00
Ivan Petkov f7c4e3cd84 process: Bump min supported rustc version to 1.25 2019-06-24 16:56:53 -07:00
Ivan Petkov 329ad3324c process: Bump to 0.2.2 2019-06-24 16:56:53 -07:00
Ivan Petkov 2b6695d25a process: Update CHANGELOG 2019-06-24 16:56:53 -07:00
Ivan Petkov 9290602815 process: Unix: preregister for signal notifications before polling child 2019-06-24 16:56:53 -07:00
Ivan Petkov 827e77e71e process: Bump to 0.2.1 2019-06-24 16:56:52 -07:00
Ivan Petkov 7b3e4b98ac process: Update Child::forget example to use the tokio runtime 2019-06-24 16:56:52 -07:00
Ivan Petkov 5e9d60e834 process: Add a CHANGELOG 2019-06-24 16:56:52 -07:00
Ivan Petkov 8270965459 process: Remove dependency on tokio-core 2019-06-24 16:56:52 -07:00
Ivan Petkov e6b044a820 process: Bump tokio-signal version to 0.2 2019-06-24 16:56:52 -07:00
Ivan Petkov de9b401457 process: Mark status_async2/StatusAsync2 as deprecated 2019-06-24 16:56:52 -07:00
Ivan Petkov ad5179b2d5 process: Remove all items deprecated in 0.1 2019-06-24 16:56:52 -07:00
Ivan Petkov 0aceba21bd process: Bump to 0.1.6 2019-06-24 16:56:52 -07:00
Ivan Petkov 09e21eceea process: Unix: mark child as reaped on kill 2019-06-24 16:56:52 -07:00
Arvid E. Picciani bdc87856f2 process: fix zombification on Drop on unix 2019-06-24 16:56:51 -07:00
Ivan Petkov 7987b64445 process: Clarify that Child::forget docs that it can leak OS resources 2019-06-24 16:56:51 -07:00
Alex Crichton 32c928b607 process: Bump to 0.1.5 2019-06-24 16:56:51 -07:00
Alex Crichton 82aeae147d process: Update dev-dependencies 2019-06-24 16:56:51 -07:00
Alex Crichton f48944c1fb process: Update winapi to 0.3 2019-06-24 16:56:51 -07:00
Ivan Petkov f0680617ee process: Fix project name typo in README 2019-06-24 16:56:51 -07:00
Alex Crichton dbc185cd3a process: Tweak travis config 2019-06-24 16:56:51 -07:00
Alex Crichton c205e2c358 process: Fix copy/paste 2019-06-24 16:56:51 -07:00
Alex Crichton acec6356ee process: Clarify wording of license information in README. 2019-06-24 16:56:51 -07:00
Alex Crichton c11eec3908 process: Bump to 0.1.4 2019-06-24 16:56:50 -07:00
Alex Crichton 69295fac1e process: Add an Errors section to status_async2 2019-06-24 16:56:50 -07:00
Ivan Petkov b9c6eb309c process: Add status_async2 as a closer analog to spawn_async 2019-06-24 16:56:50 -07:00
Ivan Petkov 56d3914675 process: Bugfix: ensure status_async closes child's stdio handles after spawning 2019-06-24 16:56:50 -07:00
Ivan Petkov 34e71fa71a process: Add must_use annotations to all futures 2019-06-24 16:56:50 -07:00
Ivan Petkov 914b803429 process: Add Debug impls for nondeprecated structs 2019-06-24 16:56:50 -07:00
Alex Crichton 4d11784b01 process: Tweak docs and macro imports 2019-06-24 16:56:50 -07:00
Michael Pankov 50cabae181 process: Add an example with reading input line-by-line 2019-06-24 16:56:50 -07:00
Alex Crichton c101e9e11d process: Use appveyor to download rustup 2019-06-24 16:56:49 -07:00
Alex Crichton 5c5f793ef0 process: Bump to 0.1.3 2019-06-24 16:56:49 -07:00
Alex Crichton 1384b31d60 process: Update to tokio-io, mio, and tokio-core changes 2019-06-24 16:56:49 -07:00
Alex Crichton 521dc94021 process: Bump to 0.1.2 2019-06-24 16:56:49 -07:00
Alex Crichton ed23a06fb1 process: Update doc urls and metadata 2019-06-24 16:56:49 -07:00
Alex Crichton 1aee22505a process: Remove caveat about tokio-signal 2019-06-24 16:56:49 -07:00
Alex Crichton 01b5bf6761 process: Use join3 instead of two joins 2019-06-24 16:56:49 -07:00
Alex Crichton 6638cbc80e process: Update README 2019-06-24 16:56:49 -07:00
Alex Crichton 22bc5e2738 process: Bump back to 0.1.1 2019-06-24 16:56:49 -07:00
Alex Crichton a0c162c0ff process: Hide compat from docs 2019-06-24 16:56:48 -07:00
Alex Crichton ca51ae9651 process: Add back in 0.1.0 compatibility layer 2019-06-24 16:56:48 -07:00
Alex Crichton f3f99b723f process: Bump to 0.2.0 2019-06-24 16:56:48 -07:00
Alex Crichton f20e7a4d2b process: Bump minimum version of tokio-core 2019-06-24 16:56:48 -07:00
Alex Crichton 4a92c4d4b6 process: Tweak drop_kills test 2019-06-24 16:56:48 -07:00
Alex Crichton 9680ecc109 process: Share init in tests 2019-06-24 16:56:48 -07:00
Alex Crichton 124391e42b process: Add a simple wait_with_output test 2019-06-24 16:56:48 -07:00
Alex Crichton 6150be189f process: Rewrite the crate with an extension trait 2019-06-24 16:56:48 -07:00
Ivan Petkov ca9586a089 process: Add documentation to public declarations 2019-06-24 16:56:47 -07:00
Ivan Petkov 89b9792931 process: Update README with crates.io info 2019-06-24 16:56:47 -07:00
Alex Crichton a0cc60153a process: Fix nightly tests 2019-06-24 16:56:47 -07:00
Alex Crichton 7f3f868b66 process: Add Windows support for stdio streams 2019-06-24 16:56:47 -07:00
Andreas Rottmann 97ebb2275c process: [WIP] Actually be non-blocking 2019-06-24 16:56:47 -07:00
Andreas Rottmann 849a5ad0b2 process: Add support for stdio streams 2019-06-24 16:56:47 -07:00
Alex Crichton b16a8613b1 process: Test on stable 2019-06-24 16:56:47 -07:00
Alex Crichton 5664660156 process: Fix tests on nightly 2019-06-24 16:56:47 -07:00
Alex Crichton 4416ea07d8 process: Update travis token 2019-06-24 16:56:47 -07:00
Alex Crichton 56222c588b process: pass --target on appveyor 2019-06-24 16:56:46 -07:00
Alex Crichton 72179d49c5 process: Update to crates.io versions of deps 2019-06-24 16:56:46 -07:00
Alex Crichton 073a1a251a process: Track tokio-core master 2019-06-24 16:56:46 -07:00
Alex Crichton 31c81faf96 process: Add appveyor to readme 2019-06-24 16:56:46 -07:00
Alex Crichton 5e68b0d51d process: Don't build on stable, start w/ beta for now 2019-06-24 16:56:46 -07:00
Alex Crichton 4bd07ac6aa process: Add metadata info 2019-06-24 16:56:46 -07:00
Alex Crichton f4f7bb232e process: Fix a test on Windows 2019-06-24 16:56:46 -07:00
Alex Crichton 413e1b78a7 process: Fix a segfault on windows 2019-06-24 16:56:46 -07:00
Alex Crichton 649fa13a15 process: Remove unused imports 2019-06-24 16:56:45 -07:00
Alex Crichton eef655f3b1 process: Add a Windows implementation 2019-06-24 16:56:45 -07:00
Alex Crichton 97508096fa process: Initial commit 2019-06-24 16:56:41 -07:00
Carl Lerche 06c473e628 Update Tokio to use std::future. (#1120)
A first pass at updating Tokio to use `std::future`.

Implementations of `Future` from the futures crate are updated to implement
`Future` from std. Implementations of `Stream` are moved to a feature flag.

This commits disables a number of crates that have not yet been updated.
2019-06-24 12:34:30 -07:00
James Gilles aa99950b9c trace: Switch benchmarks to criterion (#1163)
Extracted from #1152

This makes it possible to run benchmarks on stable + gives more statistical reliability.
2019-06-24 12:05:45 -07:00
Takanori Ishibashi aac6998c22 chore: fix url in docs (#1173) 2019-06-24 07:33:46 -04:00
Matt Bilker df2c3cd475 trace: fix debug and debug_span macro regression from #1103 (#1170)
PR #1103 accidentally changed the log level for the debug and
debug_span macros to use the INFO level instead of the DEBUG
level. This PR corrects this regression back to the intended
behavior.
2019-06-22 16:31:25 -07:00
James Gilles 36ed35c52c trace: add program-wide default dispatcher (#1152)
## Motivation

I was just trying to use tokio-trace for a greenfield project, but I was frustrated to discover that I couldn't really use it easily.

I was using the [`runtime`](https://docs.rs/runtime/0.3.0-alpha.4/runtime/) crate, which transparently spawns a thread pool executor for futures. In that thread pool, there's no way to set a tokio-trace subscriber for the duration of each thread, since you don't control the thread initialization. You *might* be able to wrap every future you spawn with a subscriber call, but that's a lot of work.

I was also confused because the documentation said that setting a subscriber in the main thread would use that subscriber for the rest of the program. That isn't the case, though -- the subscriber will be used only on the main thread, and not on worker threads, etc.

## Solution

I added a function `set_global_default`, which works similarly to the `log` crate:

```rust
tokio_trace::subscriber::set_global_default(FooSubscriber::new());
```

The global subscriber (actually a global `Dispatch`) is a `static mut` protected by an atomic; implementation is copied from the `log` crate. It is used as a fallback if a thread has no `Dispatch` currently set. This is extremely simple to use, and doesn't break any existing functionality.

Performance-wise, thread-local `Dispatch` lookup goes from ~4.5ns to ~5ns, according to the benchmarks. So, barely any runtime overhead. (Presumably there's a little compile-time overhead but idk how to measure that.) Since the atomic guard is only ever written once, it will be shared among a CPU's cores and read very cheaply.

I added some docs to partially address #1151. I also switched the tokio-trace benchmarks to criterion because the nightly benchmarks weren't compiling (missing `dyn` flags?)
2019-06-21 16:49:53 -07:00
Eliza Weisman 5925ca7720 trace: fix level_span macros not propagating parents (#1167)
Currently, when the `trace_span!`, `debug_span!`, `info_span!`,
`warn_span!`, and `error_span!` macros are invoked with an explicit
parent, a name, and zero or more fields (no target), the macros don't
pass along the explicitly provided parent when expanding to the `span!`
macro. This is likely due to an oversight on my part.

This branch fixes these macros by adding the parent into the `span!`
macro expansion. I've also added a test to catch regressions

Shoutout to @jonhoo for catching this one!

Signed-off-by: Eliza Weisman <[email protected]>
2019-06-21 11:17:06 -07:00
Max Bruckner 2ac132fb46 runtime: better error message in block_on_all on panics (#1166) 2019-06-21 11:10:58 -04:00
Hung-I Wang f9a0cb8792 timer: Implement Default for DelayQueue (#1118) 2019-06-21 10:42:52 -04:00
Igor Gnatenko 9fa6092e5a chore: Update parking_lot to 0.8 (#1078) 2019-06-21 10:42:09 -04:00
Eliza Weisman d4adeeef2f trace: Remove the AsId trait (#1145)
While we're making breaking changes to `tokio-trace`, it would be good
to get rid of the `AsId` trait. The goal of span functions that are
generic over `Span`/`Id` can be achieved without the unnecessary
complexity of defining a new trait. This would also make the API added
to `tokio_trace_core::Event` in #1109 more consistent with the
`tokio-trace::Span` API.

This branch removes `AsId` from `tokio-trace` and replaces its uses with
`impl Into<Option<Id>>` and `impl Into<Option<&'a Id>>`. While `AsRef`
might be more semantically correct for the borrowed-`Id` conversion, its
signature doesn't permit conversion into an `Option`. Implementations of
`Into<Option<Id>>` and `Into<Option<&'a Id>>` have been added for 
`tokio_trace::Span`.

This is _technically_ a breaking API change, as it changes function
signatures. However, the existing macro syntax still works as-is, and
the tests which pass `&Id`, `&Span`, and `&Option<Id>` to the span
macros all still compile after this change.

Closes #1143

Signed-off-by: Eliza Weisman <[email protected]>
2019-06-13 12:53:08 -07:00
Steven Fackler 4f6395b31c Make threadpool::Runtime methods take &self (#1140)
The runtime is inherently multi-threaded, so it's going to have to deal
with synchronization when submitting new tasks anyway. This allows a
runtime to be shared by multiple threads more easily when e.g. building
a blocking facade over a tokio-based API.
2019-06-10 12:54:27 -07:00
yanjhk 5c0b56278b Use ThreadPool's impl of spawn (#1139) 2019-06-10 11:23:12 -07:00
Eliza Weisman 41ca9a43de trace: Add shorthand syntax for local fields (#1103)
## Motivation

A common pattern in `tokio-trace` is to use the value of a local
variable as a field on a span or event. Currently, this requires code
like:
```rust
info!(foo = foo);
```
which is not particularly ergonomic given how commonly this occurs.
Struct initializers support a shorthand syntax for fields where the name
of the field is the same as a local variable, and `tokio-trace` should
as well.

## Solution

This branch adds support for syntax like
```rust
let foo = ...;
info!(foo);
```
and 
```rust
let foo = Foo {
    bar: ...,
    ...
};
info!(foo.bar)
```
to the `tokio-trace` span and event macros. This syntax also works with
the `Debug` and `Display` field shorthand.

The span macros previously used a field name with no value to indicate 
an uninitialized field. A new issue, #1138, has been opened for finding a
replacement syntax for uninitialized fields. Until then, the `tokio-trace` 
macros will no longer provide a way to create fields without values, 
although the `-core` API will continue to support this.

Closes #1062 

Signed-off-by: Eliza Weisman <[email protected]>
2019-06-09 13:16:35 -07:00
Carl Lerche 8d0f102de8 Merge branch 'v0.1.x' into merge-0.1 2019-06-05 12:28:39 -07:00
Kevin Leimkuhler 619efed28b sync: Add Sync impl for Lock (#1116)
Signed-off-by: Kevin Leimkuhler <[email protected]>
2019-06-03 11:12:28 -07:00
Carl Lerche 18ed0be851 executor: remove unnecessary APIs from Enter. (#1115) 2019-05-31 11:11:10 -07:00
Eliza Weisman 84d5a7f5a0 trace: Change Span::enter to return a guard, add Span::in_scope (#1076)
## Motivation

Currently, the primary way to use a span is to use `.enter` and pass a
closure to be executed under the span. While that is convenient in many
settings, it also comes with two decently inconvenient drawbacks:

 - It breaks control flow statements like `return`, `?`, `break`, and
   `continue`
 - It require re-indenting a potentially large chunk of code if you wish
   it to appear under a span

## Solution

This branch changes the `Span::enter` function to return a scope guard 
that exits the span when dropped, as in:
```rust
let guard = span.enter();

// code here is within the span

drop(guard);

// code here is no longer within the span
```
The method previously called `enter`, which takes a closure and 
executes it in the span's context, is now called `Span::in_scope`, and
was reimplemented on top of the new `enter` method. 

This is a breaking change to `tokio-trace` that will be part of the
upcoming 0.2 release.

Closes #1075 

Signed-off-by: Eliza Weisman <[email protected]>
2019-05-24 15:24:13 -07:00
Carl Lerche 1b498e8aa2 Fix TCP poll_hup test (#1106)
This updates tests to track a fix applied in Mio. Previously, Mio
incorrectly fired HUP events. This was due to Mio mapping `RDHUP` to
HUP. The test is updated to correctly generate a HUP event.

Additionally, HUP events will be removed from all platforms except for
Linux. This is caused by the inability to reliably map kqueue events to
the epoll HUP behavior.
2019-05-24 14:08:07 -07:00
Eliza Weisman b2c53987d9 trace: Add shorthand for field::display and field::debug (#1088)
## Motivation

In `tokio-trace`, field values may be recorded as either a subset of
Rust primitive types or as `fmt::Display` and `fmt::Debug`
implementations. Currently, `tokio-trace` provides the `field::display`
and `field::debug` functions which wrap a type with a type that
implements `Value` using the wrapped type's `fmt::Display` or
`fmt::Debug` implementation. However, importing and using these
functions adds unnecessary boilerplate. 

In #1081, @jonhoo suggested adding shorthand syntax to the macros,
similar to that used by the `slog` crate, as a solution for the
wordiness of the current API.

## Solution

This branch adds `?` and `%` sigils to field values in the span and
event macros, which expand to the `field::debug` and `field::display`
wrappers, respectively. The shorthand sigils may be used in any position
where the macros take a field value.

For example:
```rust
trace_span!("foo", my_field = ?something, ...); // shorthand for `debug`
info!(foo = %value, bar = false, ...) // shorthand for `display`
```

Adding this shorthand required a fairly large change to how field
key-value pairs are handled by the macros --- since `%foo` and `%foo`
are not valid Rust expressions, we can no longer match repeated 
`$ident = $expr` patterns, and must now match field lists as repeated
token trees. The inner helper macros for constructing `FieldSet`s and
`ValueSet`s have to parse the token trees recursively. This added a
decent chunk of complexity, but fortunately we have a large number of
compile tests for the macros and I'm quite confident that all existing
invocations will still work.

Closes #1081

Signed-off-by: Eliza Weisman <[email protected]>
2019-05-21 10:31:48 -07:00
Carl Lerche 38092010c4 Merge branch 'v0.1.x' 2019-05-14 11:50:44 -07:00
Carl Lerche cb4aea394e Update Tokio to Rust 2018 (#1082) 2019-05-14 10:27:36 -07:00
Jeehoon Kang 79d8820050 Fix link in tokio-futures/README.md (#1085)
`tokio-futures/README.md`'s link to the examples was wrong.
2019-05-10 10:19:32 -07:00
792 changed files with 45552 additions and 56075 deletions
+12 -13
View File
@@ -8,14 +8,18 @@ freebsd_instance:
task:
name: FreeBSD 12.0
env:
LOOM_MAX_DURATION: 10
LOOM_MAX_PREEMPTIONS: 2
RUSTFLAGS: -Dwarnings
setup_script:
- pkg install -y curl
- curl https://sh.rustup.rs -sSf --output rustup.sh
- sh rustup.sh -y
- sh rustup.sh -y --profile minimal --default-toolchain stable
- . $HOME/.cargo/env
- rustup target add i686-unknown-freebsd
- |
echo "~~~~ rustc --version ~~~~"
rustc --version
# Remove any existing patch statements
mv Cargo.toml Cargo.toml.bck
sed -n '/\[patch.crates-io\]/q;p' Cargo.toml.bck > Cargo.toml
@@ -27,17 +31,12 @@ task:
echo "~~~~ Cargo.toml ~~~~"
cat Cargo.toml
echo "~~~~~~~~~~~~~~~~~~~~"
cargo_cache:
folder: $HOME/.cargo/registry
test_script:
- . $HOME/.cargo/env
- cargo test --all
- (cd tokio-trace/test-log-support && cargo test)
- (cd tokio-trace/test_static_max_level_features && cargo test)
- cargo doc --all
i686_test_script:
- . $HOME/.cargo/env
- |
cargo test --all --exclude tokio-tls --exclude tokio-macros --target i686-unknown-freebsd
before_cache_script:
- rm -rf $HOME/.cargo/registry/index
- cargo doc --all --no-deps
# TODO: Re-enable
# i686_test_script:
# - . $HOME/.cargo/env
# - |
# cargo test --all --exclude tokio-tls --exclude tokio-macros --target i686-unknown-freebsd
+62 -6
View File
@@ -12,10 +12,10 @@ use your help.
This guide will help you get started. **Do not let this guide intimidate you**.
It should be considered a map to help you navigate the process.
You may also get help with contributing in the [dev channel][dev], please join
The [dev channel][dev] is available for any concerns not covered in this guide, please join
us!
[dev]: https://gitter.im/tokio-rs/dev
[dev]: https://discord.gg/6yGkFeN
## Conduct
@@ -153,8 +153,6 @@ The type level example for `tokio_timer::Timeout` provides a good example of a
documentation test:
```
/// # extern crate futures;
/// # extern crate tokio;
/// // import the `timeout` function, usually this is done
/// // with `use tokio::prelude::*`
/// use tokio::prelude::FutureExt;
@@ -192,8 +190,6 @@ If this were a documentation test for the `Timeout::new` function, then the
example would explicitly use `Timeout::new`. For example:
```
/// # extern crate futures;
/// # extern crate tokio;
/// use tokio::timer::Timeout;
/// use futures::Future;
/// use futures::sync::oneshot;
@@ -385,3 +381,63 @@ _Adapted from the [Node.js contributing guide][node]_.
[node]: https://github.com/nodejs/node/blob/master/CONTRIBUTING.md
[hiding-a-comment]: https://help.github.com/articles/managing-disruptive-comments/#hiding-a-comment
[documentation test]: https://doc.rust-lang.org/rustdoc/documentation-tests.html
## Releasing
Since the Tokio project consists of a number of crates, many of which depend on
each other, releasing new versions to crates.io can involve some complexities.
When releasing a new version of a crate, follow these steps:
1. **Ensure that the release crate has no path dependencies.** When the HEAD
version of a Tokio crate requires unreleased changes in another Tokio crate,
the crates.io dependency on the second crate will be replaced with a path
dependency. Crates with path dependencies cannot be published, so before
publishing the dependent crate, any path dependencies must also be published.
This should be done through a form of depth-first tree traversal:
1. Starting with the first path dependency in the crate to be released,
inspect the `Cargo.toml` for the dependency. If the dependency has any
path dependencies of its own, repeat this step with the first such
dependency.
2. Begin the release process for the path dependency.
3. Once the path dependency has been published to crates.io, update the
dependent crate to depend on the crates.io version.
4. When all path dependencies have been published, the dependent crate may
be published.
To verify that a crate is ready to publish, run:
```bash
bin/publish --dry-run <CRATE NAME> <CRATE VERSION>
```
2. **Update Cargo metadata.** After releasing any path dependencies, update the
`version` field in `Cargo.toml` to the new version, and the `documentation`
field to the docs.rs URL of the new version.
3. **Update other documentation links.** Update the `#![doc(html_root_url)]`
attribute in the crate's `lib.rs` and the "Documentation" link in the crate's
`README.md` to point to the docs.rs URL of the new version.
4. **Update the changelog for the crate.** Each crate in the Tokio repository
has its own `CHANGELOG.md` in that crate's subdirectory. Any changes to that
crate since the last release should be added to the changelog. Change
descriptions may be taken from the Git history, but should be edited to
ensure a consistent format, based on [Keep A Changelog][keep-a-changelog].
Other entries in that crate's changelog may also be used for reference.
5. **Perform a final audit for breaking changes.** Compare the HEAD version of
crate with the Git tag for the most recent release version. If there are any
breaking API changes, determine if those changes can be made without breaking
existing APIs. If so, resolve those issues. Otherwise, if it is necessary to
make a breaking release, update the version numbers to reflect this.
6. **Open a pull request with your changes.** Once that pull request has been
approved by a maintainer and the pull request has been merged, continue to
the next step.
7. **Release the crate.** Run the following command:
```bash
bin/publish <NAME OF CRATE> <VERSION>
```
Your editor and prompt you to edit a message for the tag. Copy the changelog
entry for that release version into your editor and close the window.
[keep-a-changelog]: https://github.com/olivierlacan/keep-a-changelog/blob/master/CHANGELOG.md
+6 -17
View File
@@ -2,24 +2,13 @@
members = [
"tokio",
"tokio-buf",
"tokio-codec",
"tokio-current-thread",
"tokio-executor",
"tokio-fs",
"tokio-futures",
"tokio-io",
"tokio-macros",
"tokio-reactor",
"tokio-signal",
"tokio-sync",
"tokio-test",
"tokio-threadpool",
"tokio-timer",
"tokio-tcp",
"tokio-tls",
"tokio-trace",
"tokio-trace/tokio-trace-core",
"tokio-udp",
"tokio-uds",
"tokio-util",
# Internal
"examples",
"tests-build",
"tests-integration",
]
+58 -95
View File
@@ -15,7 +15,7 @@ the Rust programming language. It is:
[![Crates.io][crates-badge]][crates-url]
[![MIT licensed][mit-badge]][mit-url]
[![Build Status][azure-badge]][azure-url]
[![Gitter chat][gitter-badge]][gitter-url]
[![Discord chat][discord-badge]][discord-url]
[crates-badge]: https://img.shields.io/crates/v/tokio.svg
[crates-url]: https://crates.io/crates/tokio
@@ -23,17 +23,13 @@ the Rust programming language. It is:
[mit-url]: LICENSE
[azure-badge]: https://dev.azure.com/tokio-rs/Tokio/_apis/build/status/tokio-rs.tokio?branchName=master
[azure-url]: https://dev.azure.com/tokio-rs/Tokio/_build/latest?definitionId=1&branchName=master
[gitter-badge]: https://img.shields.io/gitter/room/tokio-rs/tokio.svg
[gitter-url]: https://gitter.im/tokio-rs/tokio
[discord-badge]: https://img.shields.io/discord/500028886025895936.svg?logo=discord&style=flat-square
[discord-url]: https://discord.gg/6yGkFeN
[Website](https://tokio.rs) |
[Guides](https://tokio.rs/docs/getting-started/hello-world/) |
[API Docs](https://docs.rs/tokio/0.1.20/tokio) |
[Chat](https://gitter.im/tokio-rs/tokio)
The API docs for the master branch are published [here][master-dox].
[master-dox]: https://tokio-rs.github.io/tokio/doc/tokio/
[Guides](https://tokio.rs/docs/) |
[API Docs](https://docs.rs/tokio/latest/tokio) |
[Chat](https://discord.gg/6yGkFeN)
## Overview
@@ -42,72 +38,74 @@ asynchronous applications with the Rust programming language. At a high
level, it provides a few major components:
* A multithreaded, work-stealing based task [scheduler].
* A [reactor] backed by the operating system's event queue (epoll, kqueue,
* A reactor backed by the operating system's event queue (epoll, kqueue,
IOCP, etc...).
* Asynchronous [TCP and UDP][net] sockets.
These components provide the runtime components necessary for building
an asynchronous application.
[net]: https://docs.rs/tokio/0.1.20/tokio/net/index.html
[reactor]: https://docs.rs/tokio/0.1.20/tokio/reactor/index.html
[scheduler]: https://docs.rs/tokio/0.1.20/tokio/runtime/index.html
[net]: https://docs.rs/tokio/latest/tokio/net/index.html
[scheduler]: https://docs.rs/tokio/latest/tokio/runtime/index.html
## Example
A basic TCP echo server with Tokio:
```rust
extern crate tokio;
use tokio::prelude::*;
use tokio::io::copy;
```rust,no_run
use tokio::net::TcpListener;
use tokio::prelude::*;
fn main() {
// Bind the server's socket.
let addr = "127.0.0.1:12345".parse().unwrap();
let listener = TcpListener::bind(&addr)
.expect("unable to bind TCP listener");
#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
let mut listener = TcpListener::bind("127.0.0.1:8080").await?;
// Pull out a stream of sockets for incoming connections
let server = listener.incoming()
.map_err(|e| eprintln!("accept failed = {:?}", e))
.for_each(|sock| {
// Split up the reading and writing parts of the
// socket.
let (reader, writer) = sock.split();
loop {
let (mut socket, _) = listener.accept().await?;
// A future that echos the data and returns how
// many bytes were copied...
let bytes_copied = copy(reader, writer);
tokio::spawn(async move {
let mut buf = [0; 1024];
// ... after which we'll print what happened.
let handle_conn = bytes_copied.map(|amt| {
println!("wrote {:?} bytes", amt)
}).map_err(|err| {
eprintln!("IO error {:?}", err)
});
// In a loop, read data from the socket and write the data back.
loop {
let n = match socket.read(&mut buf).await {
// socket closed
Ok(n) if n == 0 => return,
Ok(n) => n,
Err(e) => {
eprintln!("failed to read from socket; err = {:?}", e);
return;
}
};
// Spawn the future as a concurrent task.
tokio::spawn(handle_conn)
// Write the data back
if let Err(e) = socket.write_all(&buf[0..n]).await {
eprintln!("failed to write to socket; err = {:?}", e);
return;
}
}
});
// Start the Tokio runtime
tokio::run(server);
}
}
```
More examples can be found [here](tokio/examples).
More examples can be found [here](examples). Note that the `master` branch
is currently being updated to use `async` / `await`. The examples are
not fully ported. Examples for stable Tokio can be found
[here](https://github.com/tokio-rs/tokio/tree/v0.1.x/tokio/examples).
## Getting Help
First, see if the answer to your question can be found in the [Guides] or the
[API documentation]. If the answer is not there, there is an active community in
the [Tokio Gitter channel][chat]. We would be happy to try to answer your
question. Last, if that doesn't work, try opening an [issue] with the question.
the [Tokio Discord server][chat]. We would be happy to try to answer your
question. Last, if that doesn't work, try opening an [issue] with the question.
[chat]: https://gitter.im/tokio-rs/tokio
[Guides]: https://tokio.rs/docs/
[API documentation]: https://docs.rs/tokio/latest/tokio
[chat]: https://discord.gg/6yGkFeN
[issue]: https://github.com/tokio-rs/tokio/issues/new
## Contributing
@@ -118,57 +116,22 @@ project.
[guide]: CONTRIBUTING.md
## Project layout
## Related Projects
The `tokio` crate, found at the root, is primarily intended for use by
application developers. Library authors should depend on the sub crates, which
have greater guarantees of stability.
In addition to the crates in this repository, the Tokio project also maintains
several other libraries, including:
The crates included as part of Tokio are:
* [`tracing`] (formerly `tokio-trace`): A framework for application-level
tracing and async-aware diagnostics.
* [`tokio-current-thread`]: Schedule the execution of futures on the current
thread.
* [`mio`]: A low-level, cross-platform abstraction over OS I/O APIs that powers
`tokio`.
* [`tokio-executor`]: Task execution related traits and utilities.
* [`bytes`]: Utilities for working with bytes, including efficient byte buffers.
* [`tokio-fs`]: Filesystem (and standard in / out) APIs.
* [`tokio-futures`]: Experimental `std::future::Future` and `async` / `await` support.
* [`tokio-codec`]: Utilities for encoding and decoding protocol frames.
* [`tokio-io`]: Asynchronous I/O related traits and utilities.
* [`tokio-macros`]: Macros for usage with Tokio.
* [`tokio-reactor`]: Event loop that drives I/O resources (like TCP and UDP
sockets).
* [`tokio-tcp`]: TCP bindings for use with `tokio-io` and `tokio-reactor`.
* [`tokio-threadpool`]: Schedules the execution of futures across a pool of
threads.
* [ `tokio-timer`]: Time related APIs.
* [`tokio-udp`]: UDP bindings for use with `tokio-io` and `tokio-reactor`.
* [`tokio-uds`]: Unix Domain Socket bindings for use with `tokio-io` and
`tokio-reactor`.
[`tokio-codec`]: tokio-codec
[`tokio-current-thread`]: tokio-current-thread
[`tokio-executor`]: tokio-executor
[`tokio-fs`]: tokio-fs
[`tokio-futures`]: tokio-futures
[`tokio-io`]: tokio-io
[`tokio-macros`]: tokio-macros
[`tokio-reactor`]: tokio-reactor
[`tokio-tcp`]: tokio-tcp
[`tokio-threadpool`]: tokio-threadpool
[`tokio-timer`]: tokio-timer
[`tokio-udp`]: tokio-udp
[`tokio-uds`]: tokio-uds
[`tracing`]: https://github.com/tokio-rs/tracing
[`mio`]: https://github.com/tokio-rs/mio
[`bytes`]: https://github.com/tokio-rs/bytes
## Supported Rust Versions
-2
View File
@@ -1,2 +0,0 @@
[build]
target-dir = "../target"
-49
View File
@@ -1,49 +0,0 @@
[package]
name = "examples"
edition = "2018"
version = "0.1.0"
authors = ["Carl Lerche <[email protected]>"]
license = "MIT"
# Break out of the parent workspace
[workspace]
[[bin]]
name = "chat"
path = "src/chat.rs"
[[bin]]
name = "echo_client"
path = "src/echo_client.rs"
[[bin]]
name = "echo_server"
path = "src/echo_server.rs"
[[bin]]
name = "hyper"
path = "src/hyper.rs"
[dependencies]
tokio = { version = "0.1.18", features = ["async-await-preview"] }
futures = "0.1.23"
bytes = "0.4.9"
hyper = "0.12.8"
# Avoid using crates.io for Tokio dependencies
[patch.crates-io]
tokio = { path = "../tokio" }
tokio-codec = { path = "../tokio-codec" }
tokio-current-thread = { path = "../tokio-current-thread" }
tokio-executor = { path = "../tokio-executor" }
tokio-fs = { path = "../tokio-fs" }
tokio-futures = { path = "../tokio-futures" }
tokio-io = { path = "../tokio-io" }
tokio-reactor = { path = "../tokio-reactor" }
tokio-signal = { path = "../tokio-signal" }
tokio-tcp = { path = "../tokio-tcp" }
tokio-threadpool = { path = "../tokio-threadpool" }
tokio-timer = { path = "../tokio-timer" }
tokio-tls = { path = "../tokio-tls" }
tokio-udp = { path = "../tokio-udp" }
tokio-uds = { path = "../tokio-uds" }
-5
View File
@@ -1,5 +0,0 @@
# Tokio async/await examples
These are a separate crate in order to work around some cargo bugs. It also
allows `[patch]` to be used in `Cargo.toml` to ensure the correct lib versions
are being pulled in.
-131
View File
@@ -1,131 +0,0 @@
#![feature(await_macro, async_await)]
use tokio::await;
use tokio::codec::{LinesCodec, Decoder};
use tokio::net::{TcpListener, TcpStream};
use tokio::prelude::*;
use futures::sync::mpsc;
use std::collections::HashMap;
use std::io;
use std::net::SocketAddr;
use std::sync::{Arc, Mutex};
/// Shorthand for the transmit half of the message channel.
type Tx = mpsc::UnboundedSender<String>;
struct Shared {
peers: HashMap<SocketAddr, Tx>,
}
impl Shared {
/// Create a new, empty, instance of `Shared`.
fn new() -> Self {
Shared {
peers: HashMap::new(),
}
}
}
async fn process(stream: TcpStream, state: Arc<Mutex<Shared>>) -> io::Result<()> {
let addr = stream.peer_addr().unwrap();
let mut lines = LinesCodec::new().framed(stream);
// Extract the peer's name
let name = match await!(lines.next()) {
Some(name) => name?,
None => {
// Disconnected early
return Ok(());
}
};
println!("`{}` is joining the chat", name);
let (tx, mut rx) = mpsc::unbounded();
// Register the socket
state.lock().unwrap()
.peers.insert(addr, tx);
// Split the `lines` handle into send and recv handles. This allows spawning
// separate tasks.
let (mut lines_tx, mut lines_rx) = lines.split();
// Spawn a task that receives all lines broadcasted to us from other peers
// and writes it to the client.
tokio::spawn_async(async move {
while let Some(line) = await!(rx.next()) {
let line = line.unwrap();
await!(lines_tx.send_async(line)).unwrap();
}
});
// Use the current task to read lines from the socket and broadcast them to
// other peers.
while let Some(message) = await!(lines_rx.next()) {
// TODO: Error handling
let message = message.unwrap();
let mut line = name.clone();
line.push_str(": ");
line.push_str(&message);
line.push_str("\r\n");
let state = state.lock().unwrap();
for (peer_addr, tx) in &state.peers {
if *peer_addr != addr {
// TODO: Error handling
tx.unbounded_send(line.clone()).unwrap();
}
}
}
// Remove the client from the shared state. Doing so will also result in the
// tx task to terminate.
state.lock().unwrap()
.peers.remove(&addr)
.expect("bug");
Ok(())
}
#[tokio::main]
async fn main() {
// Create the shared state. This is how all the peers communicate.
//
// The server task will hold a handle to this. For every new client, the
// `state` handle is cloned and passed into the task that processes the
// client connection.
let state = Arc::new(Mutex::new(Shared::new()));
let addr = "127.0.0.1:6142".parse().unwrap();
// Bind a TCP listener to the socket address.
//
// Note that this is the Tokio TcpListener, which is fully async.
let listener = TcpListener::bind(&addr).unwrap();
println!("server running on localhost:6142");
// Start the Tokio runtime.
let mut incoming = listener.incoming();
while let Some(stream) = await!(incoming.next()) {
let stream = match stream {
Ok(stream) => stream,
Err(_) => continue,
};
let state = state.clone();
tokio::spawn_async(async move {
if let Err(_) = await!(process(stream, state)) {
eprintln!("failed to process connection");
}
});
}
}
-50
View File
@@ -1,50 +0,0 @@
#![feature(await_macro, async_await)]
use tokio::await;
use tokio::net::TcpStream;
use tokio::prelude::*;
use std::io;
use std::net::SocketAddr;
const MESSAGES: &[&str] = &[
"hello",
"world",
"one two three",
];
async fn run_client(addr: &SocketAddr) -> io::Result<()> {
let mut stream = await!(TcpStream::connect(addr))?;
// Buffer to read into
let mut buf = [0; 128];
for msg in MESSAGES {
println!(" > write = {:?}", msg);
// Write the message to the server
await!(stream.write_all_async(msg.as_bytes()))?;
// Read the message back from the server
await!(stream.read_exact_async(&mut buf[..msg.len()]))?;
assert_eq!(&buf[..msg.len()], msg.as_bytes());
}
Ok(())
}
#[tokio::main]
async fn main() {
use std::env;
let addr = env::args().nth(1).unwrap_or("127.0.0.1:8080".to_string());
let addr = addr.parse::<SocketAddr>().unwrap();
// Connect to the echo serveer
match await!(run_client(&addr)) {
Ok(_) => println!("done."),
Err(e) => eprintln!("echo client failed; error = {:?}", e),
}
}
-42
View File
@@ -1,42 +0,0 @@
#![feature(await_macro, async_await)]
use tokio::await;
use tokio::net::{TcpListener, TcpStream};
use tokio::prelude::*;
use std::net::SocketAddr;
fn handle(mut stream: TcpStream) {
tokio::spawn_async(async move {
let mut buf = [0; 1024];
loop {
match await!(stream.read_async(&mut buf)).unwrap() {
0 => break, // Socket closed
n => {
// Send the data back
await!(stream.write_all_async(&buf[0..n])).unwrap();
}
}
}
});
}
#[tokio::main]
async fn main() {
use std::env;
let addr = env::args().nth(1).unwrap_or("127.0.0.1:8080".to_string());
let addr = addr.parse::<SocketAddr>().unwrap();
// Bind the TCP listener
let listener = TcpListener::bind(&addr).unwrap();
println!("Listening on: {}", addr);
let mut incoming = listener.incoming();
while let Some(stream) = await!(incoming.next()) {
let stream = stream.unwrap();
handle(stream);
}
}
-29
View File
@@ -1,29 +0,0 @@
#![feature(await_macro, async_await)]
use tokio::await;
use tokio::prelude::*;
use hyper::Client;
use std::time::Duration;
use std::str;
#[tokio::main]
async fn main() {
let client = Client::new();
let uri = "http://httpbin.org/ip".parse().unwrap();
let response = await!({
client.get(uri)
.timeout(Duration::from_secs(10))
}).unwrap();
println!("Response: {}", response.status());
let mut body = response.into_body();
while let Some(chunk) = await!(body.next()) {
let chunk = chunk.unwrap();
println!("chunk = {}", str::from_utf8(&chunk[..]).unwrap());
}
}
-22
View File
@@ -1,22 +0,0 @@
#![feature(await_macro, async_await)]
use tokio::await;
use tokio::timer::Delay;
use std::time::{Duration, Instant};
#[tokio::test]
async fn success_no_async() {
assert!(true);
}
#[tokio::test]
#[should_panic]
async fn fail_no_async() {
assert!(false);
}
#[tokio::test]
async fn use_timer() {
let when = Instant::now() + Duration::from_millis(10);
await!(Delay::new(when));
}
+64 -70
View File
@@ -1,88 +1,61 @@
trigger: ["master", "v0.1.x"]
pr: ["master", "v0.1.x"]
trigger: ["master"]
pr: ["master"]
variables:
RUSTFLAGS: -Dwarnings
nightly: nightly-2019-11-16
jobs:
# Check formatting
- template: ci/azure-rustfmt.yml
parameters:
name: rustfmt
# Test top level crate
- template: ci/azure-test-stable.yml
parameters:
name: test_tokio
rust: stable
displayName: Test tokio
cross: true
crates:
- tokio
- tests-integration
# Test crates that are platform specific
- template: ci/azure-test-stable.yml
parameters:
name: test_sub_cross
displayName: Test sub crates -
cross: true
crates:
- tokio-fs
- tokio-reactor
- tokio-signal
- tokio-tcp
- tokio-tls
- tokio-udp
- tokio-uds
# Test crates that are NOT platform specific
# Test sub crates
- template: ci/azure-test-stable.yml
parameters:
name: test_linux
displayName: Test sub crates -
crates:
- tokio-buf
- tokio-codec
- tokio-current-thread
- tokio-executor
- tokio-io
- tokio-sync
- tokio-threadpool
- tokio-timer
- tokio-test
- tokio-trace
- tokio-trace/tokio-trace-core
- tokio-trace/test-log-support
- tokio-trace/test_static_max_level_features
- template: ci/azure-cargo-check.yml
parameters:
name: features
displayName: Check feature permtuations
rust: stable
crates:
tokio:
- codec
- fs
- io
- reactor
- rt-full
- tcp
- timer
- udp
- uds
- sync
tokio-buf:
- util
- tokio-macros
- tokio-test
- tokio-tls
- tokio-util
- examples
# Run async-await tests
- template: ci/azure-test-nightly.yml
# Run tests from `tests-build`. This requires a different process
- template: ci/azure-test-build.yml
parameters:
name: test_nightly
displayName: Test Async / Await
rust: nightly-2019-04-25
name: test_build
displayName: Test build permutations
rust: stable
# Run loom tests
- template: ci/azure-loom.yml
parameters:
name: loom
rust: stable
crates:
- tokio
# Try cross compiling
- template: ci/azure-cross-compile.yml
parameters:
name: cross_32bit_linux
target: i686-unknown-linux-gnu
name: cross
rust: stable
# Check each feature works properly
- template: ci/azure-check-features.yml
parameters:
rust: $(nightly)
name: check_features
# This represents the minimum Rust version supported by
# Tokio. Updating this should be done in a dedicated PR and
@@ -94,21 +67,42 @@ jobs:
- template: ci/azure-check-minrust.yml
parameters:
name: minrust
rust_version: 1.26.0
rust: 1.39.0
- template: ci/azure-tsan.yml
# Check formatting
- template: ci/azure-rustfmt.yml
parameters:
name: tsan
rust: stable
name: rustfmt
# Apply clippy lints to all crates
- template: ci/azure-clippy.yml
parameters:
rust: stable
name: clippy
# Check doc generation
- template: ci/azure-check-docs.yml
parameters:
rust: $(nightly)
name: docs
# - template: ci/azure-tsan.yml
# parameters:
# name: tsan
# rust: stable
- template: ci/azure-deploy-docs.yml
parameters:
rust: stable
dependsOn:
- rustfmt
- clippy
- test_tokio
- test_sub_cross
- test_linux
- features
- test_nightly
- cross_32bit_linux
- test_build
- loom
- cross
- minrust
- tsan
- check_features
# - tsan
Executable
+121
View File
@@ -0,0 +1,121 @@
#!/usr/bin/env bash
set -e
USAGE="Publish a new release of a tokio crate
USAGE:
$(basename "$0") [OPTIONS] [CRATE] [VERSION]
OPTIONS:
-v, --verbose Use verbose Cargo output
-d, --dry-run Perform a dry run (do not publish or tag the release)
-h, --help Show this help text and exit"
DRY_RUN=""
VERBOSE=""
err() {
echo -e "\e[31m\e[1merror:\e[0m $@" 1>&2;
}
status() {
WIDTH=12
printf "\e[32m\e[1m%${WIDTH}s\e[0m %s\n" "$1" "$2"
}
verify() {
status "Verifying" "if $CRATE v$VERSION can be released"
ACTUAL=$(cargo pkgid | sed -n 's/.*#\(.*\)/\1/p')
if [ "$ACTUAL" != "$VERSION" ]; then
err "expected to release version $VERSION, but Cargo.toml contained $ACTUAL"
exit 1
fi
if git tag -l | grep -Fxq "$TAG" ; then
err "git tag \`$TAG\` already exists"
exit 1
fi
PATH_DEPS=$(grep -F "path = \"" Cargo.toml | sed -e 's/^/ /')
if [ -n "$PATH_DEPS" ]; then
err "crate \`$CRATE\` contained path dependencies:\n$PATH_DEPS"
echo "path dependencies must be removed prior to release"
exit 1
fi
}
release() {
status "Releasing" "$CRATE v$VERSION"
cargo package $VERBOSE
cargo publish $VERBOSE $DRY_RUN
status "Tagging" "$TAG"
if [ -n "$DRY_RUN" ]; then
echo "# git tag $TAG && git push --tags"
else
git tag "$TAG" && git push --tags
fi
}
while [[ $# -gt 0 ]]
do
case "$1" in
-h|--help)
echo "$USAGE"
exit 0
;;
-v|--verbose)
VERBOSE="--verbose"
set +x
shift
;;
-d|--dry-run)
DRY_RUN="--dry-run"
shift
;;
-*)
err "unknown flag \"$1\""
echo "$USAGE"
exit 1
;;
*) # crate or version
if [ -z "$CRATE" ]; then
CRATE="$1"
elif [ -z "$VERSION" ]; then
VERSION="$1"
else
err "unknown positional argument \"$1\""
echo "$USAGE"
exit 1
fi
shift
;;
esac
done
# set -- "${POSITIONAL[@]}"
if [ -z "$VERSION" ]; then
err "no version specified!"
HELP=1
fi
if [ -n "$CRATE" ]; then
TAG="$CRATE-$VERSION"
else
err "no crate specified!"
HELP=1
fi
if [ -n "$HELP" ]; then
echo "$USAGE"
exit 1
fi
if [ -d "$CRATE" ]; then
(cd "$CRATE" && verify && release )
else
err "no such crate \"$CRATE\""
exit 1
fi
Executable
+118
View File
@@ -0,0 +1,118 @@
#!/usr/bin/env bash
set -e
USAGE="Update links to docs.rs in a tokio crate
USAGE:
$(basename "$0") [OPTIONS] [CRATE] [VERSION]
OPTIONS:
-d, --dry-run Perform a dry run (do not modify any file)
-h, --help Show this help text and exit"
err() {
echo -e "\e[31m\e[1merror:\e[0m $@" 1>&2;
}
status() {
WIDTH=12
printf "\e[32m\e[1m%${WIDTH}s\e[0m %s\n" "$1" "$2"
}
c1grep() { grep "$@" || test $? = 1; }
update_versions_in_doc() {
# Print what is being/would be done
if [ -n "$DRY_RUN" ]; then
local MSG="Would change:"
else
local MSG="Updating:"
fi
git grep -lr "docs.rs/$CRATE/" \
| xargs sed --quiet \
-E "s|docs.rs/$CRATE/[0-9.]+|docs.rs/$CRATE/$VERSION|gp" \
| sed -e "s/^/$MSG /"
# Apply changes if not in dry run
if [ -z "$DRY_RUN" ]; then
git grep -lr "docs.rs/$CRATE/" \
| xargs sed -i \
-E "s|docs.rs/$CRATE/[0-9.]+|docs.rs/$CRATE/$VERSION|g"
fi
}
update() {
update_versions_in_doc
}
show_outdated() {
OUTDATED=$(git grep -rn "docs.rs/$CRATE/" \
| c1grep -v "$VERSION" \
| sed -e 's/^/ - /')
if [[ -n "$OUTDATED" ]]; then
echo "Found the following links to docs.rs with an outdated version:"
echo "$OUTDATED"
echo
else
echo "Nothing to do."
exit 1
fi
}
while [[ $# -gt 0 ]]
do
case "$1" in
-h|--help)
echo "$USAGE"
exit 0
;;
-d|--dry-run)
DRY_RUN="--dry-run"
shift
;;
-*)
err "unknown flag \"$1\""
echo "$USAGE"
exit 1
;;
*) # crate or version
if [ -z "$CRATE" ]; then
CRATE="$1"
elif [ -z "$VERSION" ]; then
VERSION="$1"
else
err "unknown positional argument \"$1\""
echo "$USAGE"
exit 1
fi
shift
;;
esac
done
# set -- "${POSITIONAL[@]}"
if [ -z "$VERSION" ]; then
err "no version specified!"
HELP=1
fi
if [ -n "$CRATE" ]; then
TAG="$CRATE-$VERSION"
else
err "no crate specified!"
HELP=1
fi
if [ -n "$HELP" ]; then
echo "$USAGE"
exit 1
fi
if [ -d "$CRATE" ]; then
# Does not cd in order to update everywhere
show_outdated && update
else
err "no such crate \"$CRATE\""
exit 1
fi
+15
View File
@@ -0,0 +1,15 @@
jobs:
# Check docs
- job: ${{ parameters.name }}
displayName: Check docs
pool:
vmImage: ubuntu-16.04
steps:
- template: azure-install-rust.yml
parameters:
rust_version: ${{ parameters.rust }}
- script: |
RUSTDOCFLAGS="--cfg docsrs" cargo doc --lib --no-deps --all-features
displayName: Check docs
+32
View File
@@ -0,0 +1,32 @@
jobs:
- job: ${{ parameters.name }}
displayName: Check features
strategy:
matrix:
Linux:
vmImage: ubuntu-16.04
MacOS:
vmImage: macOS-10.13
Windows:
vmImage: vs2017-win2016
pool:
vmImage: $(vmImage)
steps:
- template: azure-install-rust.yml
parameters:
rust_version: ${{ parameters.rust }}
- template: azure-patch-crates.yml
- script: cargo install cargo-hack
displayName: Install cargo-hack
# Check each feature works properly
# * --each-feature
# run for each feature which includes --no-default-features and default features of package
# * -Z avoid-dev-deps
# build without dev-dependencies to avoid https://github.com/rust-lang/cargo/issues/4866
# tracking-issue: https://github.com/rust-lang/cargo/issues/5133
- script: cargo hack check --all --each-feature -Z avoid-dev-deps
displayName: cargo hack check --all --each-feature
+1 -1
View File
@@ -6,7 +6,7 @@ jobs:
steps:
- template: azure-install-rust.yml
parameters:
rust_version: ${{ parameters.rust_version }}
rust_version: ${{ parameters.rust }}
- template: azure-patch-crates.yml
+16
View File
@@ -0,0 +1,16 @@
jobs:
- job: ${{ parameters.name }}
displayName: Clippy
pool:
vmImage: ubuntu-16.04
steps:
- template: azure-install-rust.yml
parameters:
rust_version: ${{ parameters.rust }}
- script: |
rustup component add clippy
cargo clippy --version
displayName: Install clippy
- script: |
cargo clippy --all --all-features -- -A clippy::mutex-atomic
displayName: cargo clippy --all
+26 -9
View File
@@ -1,27 +1,44 @@
jobs:
- job: ${{ parameters.name }}
displayName: ${{ parameters.displayName }}
strategy:
matrix:
i686:
vmImage: ubuntu-16.04
target: i686-unknown-linux-gnu
powerpc:
vmImage: ubuntu-16.04
target: powerpc-unknown-linux-gnu
powerpc64:
vmImage: ubuntu-16.04
target: powerpc64-unknown-linux-gnu
mips:
vmImage: ubuntu-16.04
target: mips-unknown-linux-gnu
arm:
vmImage: ubuntu-16.04
target: arm-linux-androideabi
pool:
vmImage: ubuntu-16.04
vmImage: $(vmImage)
steps:
- template: azure-install-rust.yml
parameters:
rust_version: stable
rust_version: ${{ parameters.rust }}
- script: sudo apt-get update
displayName: "apt-get update"
displayName: apt-get update
- script: sudo apt-get install gcc-multilib
displayName: "Install gcc-multilib"
displayName: Install gcc-multilib
- script: rustup target add ${{ parameters.target }}
displayName: "Add target"
- script: cargo install cross
displayName: Install cross
# Always patch
- template: azure-patch-crates.yml
- script: cargo check --all --exclude tokio-tls --target ${{ parameters.target }}
- script: cross check --all --exclude tokio-tls --target $(target)
displayName: Check source
- script: cargo check --tests --all --exclude tokio-tls --target ${{ parameters.target }}
displayName: Check tests
# - script: cross check --tests --all --exclude tokio-tls --target $(target)
# displayName: Check tests
+3 -2
View File
@@ -12,9 +12,10 @@ jobs:
steps:
- template: azure-install-rust.yml
parameters:
rust_version: stable
# rust_version: stable
rust_version: ${{ parameters.rust }}
- script: |
cargo doc --all --no-deps
cargo doc --all --no-deps --all-features
cp -R target/doc '$(Build.BinariesDirectory)'
displayName: 'Generate Documentation'
- script: |
+4 -4
View File
@@ -2,7 +2,7 @@ steps:
# Linux and macOS.
- script: |
set -e
curl https://sh.rustup.rs -sSf | sh -s -- -y --default-toolchain none
curl https://sh.rustup.rs -sSf | sh -s -- -y --profile minimal --default-toolchain none
export PATH=$PATH:$HOME/.cargo/bin
rustup toolchain install $RUSTUP_TOOLCHAIN
rustup default $RUSTUP_TOOLCHAIN
@@ -14,20 +14,20 @@ steps:
# Windows.
- script: |
echo "windows"
curl -sSf -o rustup-init.exe https://win.rustup.rs
rustup-init.exe -y --default-toolchain none
rustup-init.exe -y --profile minimal --default-toolchain none
set PATH=%PATH%;%USERPROFILE%\.cargo\bin
rustup toolchain install %RUSTUP_TOOLCHAIN%
rustup default %RUSTUP_TOOLCHAIN%
echo "##vso[task.setvariable variable=PATH;]%PATH%;%USERPROFILE%\.cargo\bin"
env:
RUSTUP_TOOLCHAIN: ${{parameters.rust_version}}
displayName: Install rust (windows)
displayName: "Install rust (windows)"
condition: eq(variables['Agent.OS'], 'Windows_NT')
# All platforms.
- script: |
rustup toolchain list
rustc -Vv
cargo -V
displayName: Query rust and cargo versions
+18
View File
@@ -0,0 +1,18 @@
jobs:
- job: ${{ parameters.name }}
displayName: Loom tests
pool:
vmImage: ubuntu-16.04
steps:
- template: azure-install-rust.yml
parameters:
rust_version: ${{ parameters.rust }}
- ${{ each crate in parameters.crates }}:
- script: RUSTFLAGS="--cfg loom" cargo test --lib --release --features "full" -- --test-threads=1 --nocapture
env:
LOOM_MAX_PREEMPTIONS: 1
CI: 'True'
displayName: test ${{ crate }}
workingDirectory: $(Build.SourcesDirectory)/${{ crate }}
+2 -1
View File
@@ -7,9 +7,10 @@ jobs:
steps:
- template: azure-install-rust.yml
parameters:
rust_version: stable
rust_version: ${{ parameters.rust }}
- script: |
rustup component add rustfmt
cargo fmt --version
displayName: Install rustfmt
- script: |
cargo fmt --all -- --check
+17
View File
@@ -0,0 +1,17 @@
jobs:
- job: ${{ parameters.name }}
displayName: ${{ parameters.displayName }}
pool:
vmImage: 'Ubuntu 16.04'
steps:
- template: azure-install-rust.yml
parameters:
rust_version: ${{ parameters.rust }}
- script: cargo install cargo-hack
displayName: Install cargo-hack
- script: cargo hack test --each-feature
displayName: cargo hack test --each-feature
workingDirectory: $(Build.SourcesDirectory)/tests-build
+9 -8
View File
@@ -17,25 +17,26 @@ jobs:
steps:
- template: azure-install-rust.yml
parameters:
rust_version: stable
rust_version: ${{ parameters.rust }}
- template: azure-is-release.yml
- ${{ each crate in parameters.crates }}:
- script: cargo test
# Run with all crate features
- script: cargo test --all-features
env:
LOOM_MAX_DURATION: 10
LOOM_MAX_PREEMPTIONS: 2
CI: 'True'
displayName: cargo test -p ${{ crate }}
displayName: ${{ crate }} - cargo test --all-features
workingDirectory: $(Build.SourcesDirectory)/${{ crate }}
condition: and(succeeded(), ne(variables['isRelease'], 'true'))
- template: azure-patch-crates.yml
- ${{ each crate in parameters.crates }}:
- script: cargo test
# Run with all crate features
- script: cargo test --all-features
env:
LOOM_MAX_DURATION: 10
LOOM_MAX_PREEMPTIONS: 2
CI: 'True'
displayName: cargo test -p ${{ crate }} (PATCHED)
displayName: ${{ crate }} - cargo test --all-features
workingDirectory: $(Build.SourcesDirectory)/${{ crate }}
+1 -3
View File
@@ -5,14 +5,12 @@ jobs:
matrix:
Timer:
cmd: cargo test -p tokio-timer --test hammer
Threadpool:
cmd: cargo test -p tokio-threadpool --tests
pool:
vmImage: ubuntu-16.04
steps:
- template: azure-install-rust.yml
parameters:
rust_version: nightly-2018-11-18
rust_version: ${{ parameters.rust }}
- template: azure-patch-crates.yml
- script: |
+3 -17
View File
@@ -2,21 +2,7 @@
# repository.
[patch.crates-io]
tokio = { path = "tokio" }
tokio-buf = { path = "tokio-buf" }
tokio-codec = { path = "tokio-codec" }
tokio-current-thread = { path = "tokio-current-thread" }
tokio-executor = { path = "tokio-executor" }
tokio-fs = { path = "tokio-fs" }
tokio-futures = { path = "tokio-futures" }
tokio-io = { path = "tokio-io" }
tokio-reactor = { path = "tokio-reactor" }
tokio-signal = { path = "tokio-signal" }
tokio-sync = { path = "tokio-sync" }
tokio-threadpool = { path = "tokio-threadpool" }
tokio-timer = { path = "tokio-timer" }
tokio-tcp = { path = "tokio-tcp" }
tokio-macros = { path = "tokio-macros" }
tokio-test = { path = "tokio-test" }
tokio-tls = { path = "tokio-tls" }
tokio-trace = { path = "tokio-trace" }
tokio-trace-core = { path = "tokio-trace/tokio-trace-core" }
tokio-udp = { path = "tokio-udp" }
tokio-uds = { path = "tokio-uds" }
tokio-util = { path = "tokio-util" }
+2
View File
@@ -8,6 +8,8 @@ race:Weak*drop
# `std` mpsc is not used in any Tokio code base. This race is triggered by some
# rust runtime logic.
race:std*mpsc_queue
race:std*lang_start
race:drop*std::thread*
# Probably more fences in std.
race:__call_tls_dtors
+61
View File
@@ -0,0 +1,61 @@
[package]
name = "examples"
version = "0.0.0"
publish = false
edition = "2018"
[dev-dependencies]
tokio = { version = "0.2.0", path = "../tokio", features = ["full"] }
tokio-util = { version = "0.2.0", path = "../tokio-util", features = ["full"] }
bytes = "0.5"
futures = "0.3.0"
http = "0.2"
serde = "1.0"
serde_derive = "1.0"
serde_json = "1.0"
httparse = "1.0"
time = "0.1"
[[example]]
name = "chat"
path = "chat.rs"
[[example]]
name = "connect"
path = "connect.rs"
[[example]]
name = "echo-udp"
path = "echo-udp.rs"
[[example]]
name = "echo"
path = "echo.rs"
[[example]]
name = "hello_world"
path = "hello_world.rs"
[[example]]
name = "print_each_packet"
path = "print_each_packet.rs"
[[example]]
name = "proxy"
path = "proxy.rs"
[[example]]
name = "tinydb"
path = "tinydb.rs"
[[example]]
name = "udp-client"
path = "udp-client.rs"
[[example]]
name = "udp-codec"
path = "udp-codec.rs"
[[example]]
name = "tinyhttp"
path = "tinyhttp.rs"
+6
View File
@@ -0,0 +1,6 @@
## Examples of how to use Tokio
The `master` branch is currently being updated to use `async` / `await`.
The examples are not fully ported. Examples for stable Tokio can be
found
[here](https://github.com/tokio-rs/tokio/tree/v0.1.x/tokio/examples).
+255
View File
@@ -0,0 +1,255 @@
//! A chat server that broadcasts a message to all connections.
//!
//! This example is explicitly more verbose than it has to be. This is to
//! illustrate more concepts.
//!
//! A chat server for telnet clients. After a telnet client connects, the first
//! line should contain the client's name. After that, all lines sent by a
//! client are broadcasted to all other connected clients.
//!
//! Because the client is telnet, lines are delimited by "\r\n".
//!
//! You can test this out by running:
//!
//! cargo run --example chat
//!
//! And then in another terminal run:
//!
//! telnet localhost 6142
//!
//! You can run the `telnet` command in any number of additional windows.
//!
//! You can run the second command in multiple windows and then chat between the
//! two, seeing the messages from the other client as they're received. For all
//! connected clients they'll all join the same room and see everyone else's
//! messages.
#![warn(rust_2018_idioms)]
use tokio::net::{TcpListener, TcpStream};
use tokio::sync::{mpsc, Mutex};
use tokio_util::codec::{Framed, LinesCodec, LinesCodecError};
use futures::{SinkExt, Stream, StreamExt};
use std::collections::HashMap;
use std::env;
use std::error::Error;
use std::io;
use std::net::SocketAddr;
use std::pin::Pin;
use std::sync::Arc;
use std::task::{Context, Poll};
#[tokio::main]
async fn main() -> Result<(), Box<dyn Error>> {
// Create the shared state. This is how all the peers communicate.
//
// The server task will hold a handle to this. For every new client, the
// `state` handle is cloned and passed into the task that processes the
// client connection.
let state = Arc::new(Mutex::new(Shared::new()));
let addr = env::args().nth(1).unwrap_or("127.0.0.1:6142".to_string());
// Bind a TCP listener to the socket address.
//
// Note that this is the Tokio TcpListener, which is fully async.
let mut listener = TcpListener::bind(&addr).await?;
println!("server running on {}", addr);
loop {
// Asynchronously wait for an inbound TcpStream.
let (stream, addr) = listener.accept().await?;
// Clone a handle to the `Shared` state for the new connection.
let state = Arc::clone(&state);
// Spawn our handler to be run asynchronously.
tokio::spawn(async move {
if let Err(e) = process(state, stream, addr).await {
println!("an error occured; error = {:?}", e);
}
});
}
}
/// Shorthand for the transmit half of the message channel.
type Tx = mpsc::UnboundedSender<String>;
/// Shorthand for the receive half of the message channel.
type Rx = mpsc::UnboundedReceiver<String>;
/// Data that is shared between all peers in the chat server.
///
/// This is the set of `Tx` handles for all connected clients. Whenever a
/// message is received from a client, it is broadcasted to all peers by
/// iterating over the `peers` entries and sending a copy of the message on each
/// `Tx`.
struct Shared {
peers: HashMap<SocketAddr, Tx>,
}
/// The state for each connected client.
struct Peer {
/// The TCP socket wrapped with the `Lines` codec, defined below.
///
/// This handles sending and receiving data on the socket. When using
/// `Lines`, we can work at the line level instead of having to manage the
/// raw byte operations.
lines: Framed<TcpStream, LinesCodec>,
/// Receive half of the message channel.
///
/// This is used to receive messages from peers. When a message is received
/// off of this `Rx`, it will be written to the socket.
rx: Rx,
}
impl Shared {
/// Create a new, empty, instance of `Shared`.
fn new() -> Self {
Shared {
peers: HashMap::new(),
}
}
/// Send a `LineCodec` encoded message to every peer, except
/// for the sender.
async fn broadcast(&mut self, sender: SocketAddr, message: &str) {
for peer in self.peers.iter_mut() {
if *peer.0 != sender {
let _ = peer.1.send(message.into());
}
}
}
}
impl Peer {
/// Create a new instance of `Peer`.
async fn new(
state: Arc<Mutex<Shared>>,
lines: Framed<TcpStream, LinesCodec>,
) -> io::Result<Peer> {
// Get the client socket address
let addr = lines.get_ref().peer_addr()?;
// Create a channel for this peer
let (tx, rx) = mpsc::unbounded_channel();
// Add an entry for this `Peer` in the shared state map.
state.lock().await.peers.insert(addr, tx);
Ok(Peer { lines, rx })
}
}
#[derive(Debug)]
enum Message {
/// A message that should be broadcasted to others.
Broadcast(String),
/// A message that should be received by a client
Received(String),
}
// Peer implements `Stream` in a way that polls both the `Rx`, and `Framed` types.
// A message is produced whenever an event is ready until the `Framed` stream returns `None`.
impl Stream for Peer {
type Item = Result<Message, LinesCodecError>;
fn poll_next(mut self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll<Option<Self::Item>> {
// First poll the `UnboundedReceiver`.
if let Poll::Ready(Some(v)) = self.rx.poll_next_unpin(cx) {
return Poll::Ready(Some(Ok(Message::Received(v))));
}
// Secondly poll the `Framed` stream.
let result: Option<_> = futures::ready!(self.lines.poll_next_unpin(cx));
Poll::Ready(match result {
// We've received a message we should broadcast to others.
Some(Ok(message)) => Some(Ok(Message::Broadcast(message))),
// An error occured.
Some(Err(e)) => Some(Err(e)),
// The stream has been exhausted.
None => None,
})
}
}
/// Process an individual chat client
async fn process(
state: Arc<Mutex<Shared>>,
stream: TcpStream,
addr: SocketAddr,
) -> Result<(), Box<dyn Error>> {
let mut lines = Framed::new(stream, LinesCodec::new());
// Send a prompt to the client to enter their username.
lines
.send(String::from("Please enter your username:"))
.await?;
// Read the first line from the `LineCodec` stream to get the username.
let username = match lines.next().await {
Some(Ok(line)) => line,
// We didn't get a line so we return early here.
_ => {
println!("Failed to get username from {}. Client disconnected.", addr);
return Ok(());
}
};
// Register our peer with state which internally sets up some channels.
let mut peer = Peer::new(state.clone(), lines).await?;
// A client has connected, let's let everyone know.
{
let mut state = state.lock().await;
let msg = format!("{} has joined the chat", username);
println!("{}", msg);
state.broadcast(addr, &msg).await;
}
// Process incoming messages until our stream is exhausted by a disconnect.
while let Some(result) = peer.next().await {
match result {
// A message was received from the current user, we should
// broadcast this message to the other users.
Ok(Message::Broadcast(msg)) => {
let mut state = state.lock().await;
let msg = format!("{}: {}", username, msg);
state.broadcast(addr, &msg).await;
}
// A message was received from a peer. Send it to the
// current user.
Ok(Message::Received(msg)) => {
peer.lines.send(msg).await?;
}
Err(e) => {
println!(
"an error occured while processing messages for {}; error = {:?}",
username, e
);
}
}
}
// If this section is reached it means that the client was disconnected!
// Let's let everyone still connected know about it.
{
let mut state = state.lock().await;
state.peers.remove(&addr);
let msg = format!("{} has left the chat", username);
println!("{}", msg);
state.broadcast(addr, &msg).await;
}
Ok(())
}
+184
View File
@@ -0,0 +1,184 @@
//! An example of hooking up stdin/stdout to either a TCP or UDP stream.
//!
//! This example will connect to a socket address specified in the argument list
//! and then forward all data read on stdin to the server, printing out all data
//! received on stdout. An optional `--udp` argument can be passed to specify
//! that the connection should be made over UDP instead of TCP, translating each
//! line entered on stdin to a UDP packet to be sent to the remote address.
//!
//! Note that this is not currently optimized for performance, especially
//! around buffer management. Rather it's intended to show an example of
//! working with a client.
//!
//! This example can be quite useful when interacting with the other examples in
//! this repository! Many of them recommend running this as a simple "hook up
//! stdin/stdout to a server" to get up and running.
#![warn(rust_2018_idioms)]
use tokio::io;
use tokio_util::codec::{FramedRead, FramedWrite};
use std::env;
use std::error::Error;
use std::net::SocketAddr;
#[tokio::main]
async fn main() -> Result<(), Box<dyn Error>> {
// Determine if we're going to run in TCP or UDP mode
let mut args = env::args().skip(1).collect::<Vec<_>>();
let tcp = match args.iter().position(|a| a == "--udp") {
Some(i) => {
args.remove(i);
false
}
None => true,
};
// Parse what address we're going to connect to
let addr = match args.first() {
Some(addr) => addr,
None => Err("this program requires at least one argument")?,
};
let addr = addr.parse::<SocketAddr>()?;
let stdin = FramedRead::new(io::stdin(), codec::Bytes);
let stdout = FramedWrite::new(io::stdout(), codec::Bytes);
if tcp {
tcp::connect(&addr, stdin, stdout).await?;
} else {
udp::connect(&addr, stdin, stdout).await?;
}
Ok(())
}
mod tcp {
use super::codec;
use futures::{future, Sink, SinkExt, Stream, StreamExt};
use std::{error::Error, io, net::SocketAddr};
use tokio::net::TcpStream;
use tokio_util::codec::{FramedRead, FramedWrite};
pub async fn connect(
addr: &SocketAddr,
stdin: impl Stream<Item = Result<Vec<u8>, io::Error>> + Unpin,
mut stdout: impl Sink<Vec<u8>, Error = io::Error> + Unpin,
) -> Result<(), Box<dyn Error>> {
let mut stream = TcpStream::connect(addr).await?;
let (r, w) = stream.split();
let sink = FramedWrite::new(w, codec::Bytes);
let mut stream = FramedRead::new(r, codec::Bytes)
.filter_map(|i| match i {
Ok(i) => future::ready(Some(i)),
Err(e) => {
println!("failed to read from socket; error={}", e);
future::ready(None)
}
})
.map(Ok);
match future::join(stdin.forward(sink), stdout.send_all(&mut stream)).await {
(Err(e), _) | (_, Err(e)) => Err(e.into()),
_ => Ok(()),
}
}
}
mod udp {
use tokio::net::udp::{RecvHalf, SendHalf};
use tokio::net::UdpSocket;
use futures::{future, Sink, SinkExt, Stream, StreamExt};
use std::error::Error;
use std::io;
use std::net::SocketAddr;
pub async fn connect(
addr: &SocketAddr,
stdin: impl Stream<Item = Result<Vec<u8>, io::Error>> + Unpin,
stdout: impl Sink<Vec<u8>, Error = io::Error> + Unpin,
) -> Result<(), Box<dyn Error>> {
// We'll bind our UDP socket to a local IP/port, but for now we
// basically let the OS pick both of those.
let bind_addr = if addr.ip().is_ipv4() {
"0.0.0.0:0"
} else {
"[::]:0"
};
let socket = UdpSocket::bind(&bind_addr).await?;
socket.connect(addr).await?;
let (mut r, mut w) = socket.split();
future::try_join(send(stdin, &mut w), recv(stdout, &mut r)).await?;
Ok(())
}
async fn send(
mut stdin: impl Stream<Item = Result<Vec<u8>, io::Error>> + Unpin,
writer: &mut SendHalf,
) -> Result<(), io::Error> {
while let Some(item) = stdin.next().await {
let buf = item?;
writer.send(&buf[..]).await?;
}
Ok(())
}
async fn recv(
mut stdout: impl Sink<Vec<u8>, Error = io::Error> + Unpin,
reader: &mut RecvHalf,
) -> Result<(), io::Error> {
loop {
let mut buf = vec![0; 1024];
let n = reader.recv(&mut buf[..]).await?;
if n > 0 {
stdout.send(buf).await?;
}
}
}
}
mod codec {
use bytes::{BufMut, BytesMut};
use std::io;
use tokio_util::codec::{Decoder, Encoder};
/// A simple `Codec` implementation that just ships bytes around.
///
/// This type is used for "framing" a TCP/UDP stream of bytes but it's really
/// just a convenient method for us to work with streams/sinks for now.
/// This'll just take any data read and interpret it as a "frame" and
/// conversely just shove data into the output location without looking at
/// it.
pub struct Bytes;
impl Decoder for Bytes {
type Item = Vec<u8>;
type Error = io::Error;
fn decode(&mut self, buf: &mut BytesMut) -> io::Result<Option<Vec<u8>>> {
if buf.len() > 0 {
let len = buf.len();
Ok(Some(buf.split_to(len).into_iter().collect()))
} else {
Ok(None)
}
}
}
impl Encoder for Bytes {
type Item = Vec<u8>;
type Error = io::Error;
fn encode(&mut self, data: Vec<u8>, buf: &mut BytesMut) -> io::Result<()> {
buf.put(&data[..]);
Ok(())
}
}
}
@@ -10,17 +10,13 @@
//!
//! Each line you type in to the `nc` terminal should be echo'd back to you!
#![deny(warnings)]
#[macro_use]
extern crate futures;
extern crate tokio;
#![warn(rust_2018_idioms)]
use std::error::Error;
use std::net::SocketAddr;
use std::{env, io};
use tokio;
use tokio::net::UdpSocket;
use tokio::prelude::*;
struct Server {
socket: UdpSocket,
@@ -28,47 +24,46 @@ struct Server {
to_send: Option<(usize, SocketAddr)>,
}
impl Future for Server {
type Item = ();
type Error = io::Error;
impl Server {
async fn run(self) -> Result<(), io::Error> {
let Server {
mut socket,
mut buf,
mut to_send,
} = self;
fn poll(&mut self) -> Poll<(), io::Error> {
loop {
// First we check to see if there's a message we need to echo back.
// If so then we try to send it back to the original source, waiting
// until it's writable and we're able to do so.
if let Some((size, peer)) = self.to_send {
let amt = try_ready!(self.socket.poll_send_to(&self.buf[..size], &peer));
if let Some((size, peer)) = to_send {
let amt = socket.send_to(&buf[..size], &peer).await?;
println!("Echoed {}/{} bytes to {}", amt, size, peer);
self.to_send = None;
}
// If we're here then `to_send` is `None`, so we take a look for the
// next message we're going to echo back.
self.to_send = Some(try_ready!(self.socket.poll_recv_from(&mut self.buf)));
to_send = Some(socket.recv_from(&mut buf).await?);
}
}
}
fn main() -> Result<(), Box<std::error::Error>> {
#[tokio::main]
async fn main() -> Result<(), Box<dyn Error>> {
let addr = env::args().nth(1).unwrap_or("127.0.0.1:8080".to_string());
let addr = addr.parse::<SocketAddr>()?;
let socket = UdpSocket::bind(&addr)?;
let socket = UdpSocket::bind(&addr).await?;
println!("Listening on: {}", socket.local_addr()?);
let server = Server {
socket: socket,
socket,
buf: vec![0; 1024],
to_send: None,
};
// This starts the server task.
//
// `map_err` handles the error by logging it and maps the future to a type
// that can be spawned.
//
// `tokio::run` spawns the task on the Tokio runtime and starts running.
tokio::run(server.map_err(|e| println!("server error = {:?}", e)));
server.run().await?;
Ok(())
}
+77
View File
@@ -0,0 +1,77 @@
//! A "hello world" echo server with Tokio
//!
//! This server will create a TCP listener, accept connections in a loop, and
//! write back everything that's read off of each TCP connection.
//!
//! Because the Tokio runtime uses a thread pool, each TCP connection is
//! processed concurrently with all other TCP connections across multiple
//! threads.
//!
//! To see this server in action, you can run this in one terminal:
//!
//! cargo run --example echo
//!
//! and in another terminal you can run:
//!
//! cargo run --example connect 127.0.0.1:8080
//!
//! Each line you type in to the `connect` terminal should be echo'd back to
//! you! If you open up multiple terminals running the `connect` example you
//! should be able to see them all make progress simultaneously.
#![warn(rust_2018_idioms)]
use tokio;
use tokio::io::{AsyncReadExt, AsyncWriteExt};
use tokio::net::TcpListener;
use std::env;
use std::error::Error;
#[tokio::main]
async fn main() -> Result<(), Box<dyn Error>> {
// Allow passing an address to listen on as the first argument of this
// program, but otherwise we'll just set up our TCP listener on
// 127.0.0.1:8080 for connections.
let addr = env::args().nth(1).unwrap_or("127.0.0.1:8080".to_string());
// Next up we create a TCP listener which will listen for incoming
// connections. This TCP listener is bound to the address we determined
// above and must be associated with an event loop.
let mut listener = TcpListener::bind(&addr).await?;
println!("Listening on: {}", addr);
loop {
// Asynchronously wait for an inbound socket.
let (mut socket, _) = listener.accept().await?;
// And this is where much of the magic of this server happens. We
// crucially want all clients to make progress concurrently, rather than
// blocking one on completion of another. To achieve this we use the
// `tokio::spawn` function to execute the work in the background.
//
// Essentially here we're executing a new task to run concurrently,
// which will allow all of our clients to be processed concurrently.
tokio::spawn(async move {
let mut buf = [0; 1024];
// In a loop, read data from the socket and write the data back.
loop {
let n = socket
.read(&mut buf)
.await
.expect("failed to read data from socket");
if n == 0 {
return;
}
socket
.write_all(&buf[0..n])
.await
.expect("failed to write data to socket");
}
});
}
}
+33
View File
@@ -0,0 +1,33 @@
//! Hello world server.
//!
//! A simple client that opens a TCP stream, writes "hello world\n", and closes
//! the connection.
//!
//! You can test this out by running:
//!
//! ncat -l 6142
//!
//! And then in another terminal run:
//!
//! cargo run --example hello_world
#![warn(rust_2018_idioms)]
use tokio::io::AsyncWriteExt;
use tokio::net::TcpStream;
use std::error::Error;
#[tokio::main]
pub async fn main() -> Result<(), Box<dyn Error>> {
// Open a TCP stream to the socket address.
//
// Note that this is the Tokio TcpStream, which is fully async.
let mut stream = TcpStream::connect("127.0.0.1:6142").await?;
println!("created stream");
let result = stream.write(b"hello world\n").await;
println!("wrote to stream; success={:?}", result.is_ok());
Ok(())
}
+104
View File
@@ -0,0 +1,104 @@
//! A "print-each-packet" server with Tokio
//!
//! This server will create a TCP listener, accept connections in a loop, and
//! put down in the stdout everything that's read off of each TCP connection.
//!
//! Because the Tokio runtime uses a thread pool, each TCP connection is
//! processed concurrently with all other TCP connections across multiple
//! threads.
//!
//! To see this server in action, you can run this in one terminal:
//!
//! cargo run --example print\_each\_packet
//!
//! and in another terminal you can run:
//!
//! cargo run --example connect 127.0.0.1:8080
//!
//! Each line you type in to the `connect` terminal should be written to terminal!
//!
//! Minimal js example:
//!
//! ```js
//! var net = require("net");
//!
//! var listenPort = 8080;
//!
//! var server = net.createServer(function (socket) {
//! socket.on("data", function (bytes) {
//! console.log("bytes", bytes);
//! });
//!
//! socket.on("end", function() {
//! console.log("Socket received FIN packet and closed connection");
//! });
//! socket.on("error", function (error) {
//! console.log("Socket closed with error", error);
//! });
//!
//! socket.on("close", function (with_error) {
//! if (with_error) {
//! console.log("Socket closed with result: Err(SomeError)");
//! } else {
//! console.log("Socket closed with result: Ok(())");
//! }
//! });
//!
//! });
//!
//! server.listen(listenPort);
//!
//! console.log("Listening on:", listenPort);
//! ```
//!
#![warn(rust_2018_idioms)]
use tokio::net::TcpListener;
use tokio_util::codec::{BytesCodec, Decoder};
use futures::StreamExt;
use std::env;
#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
// Allow passing an address to listen on as the first argument of this
// program, but otherwise we'll just set up our TCP listener on
// 127.0.0.1:8080 for connections.
let addr = env::args().nth(1).unwrap_or("127.0.0.1:8080".to_string());
// Next up we create a TCP listener which will listen for incoming
// connections. This TCP listener is bound to the address we determined
// above and must be associated with an event loop, so we pass in a handle
// to our event loop. After the socket's created we inform that we're ready
// to go and start accepting connections.
let mut listener = TcpListener::bind(&addr).await?;
println!("Listening on: {}", addr);
loop {
// Asynchronously wait for an inbound socket.
let (socket, _) = listener.accept().await?;
// And this is where much of the magic of this server happens. We
// crucially want all clients to make progress concurrently, rather than
// blocking one on completion of another. To achieve this we use the
// `tokio::spawn` function to execute the work in the background.
//
// Essentially here we're executing a new task to run concurrently,
// which will allow all of our clients to be processed concurrently.
tokio::spawn(async move {
// We're parsing each socket with the `BytesCodec` included in `tokio::codec`.
let mut framed = BytesCodec::new().framed(socket);
// We loop while there are messages coming from the Stream `framed`.
// The stream will return None once the client disconnects.
while let Some(message) = framed.next().await {
match message {
Ok(bytes) => println!("bytes: {:?}", bytes),
Err(err) => println!("Socket closed with error: {:?}", err),
}
}
println!("Socket received FIN packet and closed connection");
});
}
}
+68
View File
@@ -0,0 +1,68 @@
//! A proxy that forwards data to another server and forwards that server's
//! responses back to clients.
//!
//! Because the Tokio runtime uses a thread pool, each TCP connection is
//! processed concurrently with all other TCP connections across multiple
//! threads.
//!
//! You can showcase this by running this in one terminal:
//!
//! cargo run --example proxy
//!
//! This in another terminal
//!
//! cargo run --example echo
//!
//! And finally this in another terminal
//!
//! cargo run --example connect 127.0.0.1:8081
//!
//! This final terminal will connect to our proxy, which will in turn connect to
//! the echo server, and you'll be able to see data flowing between them.
#![warn(rust_2018_idioms)]
use tokio::io;
use tokio::net::{TcpListener, TcpStream};
use futures::future::try_join;
use futures::FutureExt;
use std::env;
use std::error::Error;
#[tokio::main]
async fn main() -> Result<(), Box<dyn Error>> {
let listen_addr = env::args().nth(1).unwrap_or("127.0.0.1:8081".to_string());
let server_addr = env::args().nth(2).unwrap_or("127.0.0.1:8080".to_string());
println!("Listening on: {}", listen_addr);
println!("Proxying to: {}", server_addr);
let mut listener = TcpListener::bind(listen_addr).await?;
while let Ok((inbound, _)) = listener.accept().await {
let transfer = transfer(inbound, server_addr.clone()).map(|r| {
if let Err(e) = r {
println!("Failed to transfer; error={}", e);
}
});
tokio::spawn(transfer);
}
Ok(())
}
async fn transfer(mut inbound: TcpStream, proxy_addr: String) -> Result<(), Box<dyn Error>> {
let mut outbound = TcpStream::connect(proxy_addr).await?;
let (mut ri, mut wi) = inbound.split();
let (mut ro, mut wo) = outbound.split();
let client_to_server = io::copy(&mut ri, &mut wo);
let server_to_client = io::copy(&mut ro, &mut wi);
try_join(client_to_server, server_to_client).await?;
Ok(())
}
+72 -75
View File
@@ -39,20 +39,17 @@
//! * `SET $key $value` - this will set the value of `$key` to `$value`,
//! returning the previous value, if any.
#![deny(warnings)]
#![warn(rust_2018_idioms)]
extern crate tokio;
use tokio::net::TcpListener;
use tokio_util::codec::{Framed, LinesCodec};
use futures::{SinkExt, StreamExt};
use std::collections::HashMap;
use std::env;
use std::io::BufReader;
use std::net::SocketAddr;
use std::error::Error;
use std::sync::{Arc, Mutex};
use tokio::io::{lines, write_all};
use tokio::net::TcpListener;
use tokio::prelude::*;
/// The in-memory database shared amongst all clients.
///
/// This database will be shared via `Arc`, so to mutate the internal map we're
@@ -83,12 +80,13 @@ enum Response {
},
}
fn main() -> Result<(), Box<std::error::Error>> {
#[tokio::main]
async fn main() -> Result<(), Box<dyn Error>> {
// Parse the address we're going to run this server on
// and set up our TCP listener to accept connections.
let addr = env::args().nth(1).unwrap_or("127.0.0.1:8080".to_string());
let addr = addr.parse::<SocketAddr>()?;
let listener = TcpListener::bind(&addr).map_err(|_| "failed to bind")?;
let mut listener = TcpListener::bind(&addr).await?;
println!("Listening on: {}", addr);
// Create the shared state of this server that will be shared amongst all
@@ -102,78 +100,77 @@ fn main() -> Result<(), Box<std::error::Error>> {
map: Mutex::new(initial_db),
});
let done = listener
.incoming()
.map_err(|e| println!("error accepting socket; error = {:?}", e))
.for_each(move |socket| {
// As with many other small examples, the first thing we'll do is
// *split* this TCP stream into two separately owned halves. This'll
// allow us to work with the read and write halves independently.
let (reader, writer) = socket.split();
loop {
match listener.accept().await {
Ok((socket, _)) => {
// After getting a new connection first we see a clone of the database
// being created, which is creating a new reference for this connected
// client to use.
let db = db.clone();
// Since our protocol is line-based we use `tokio_io`'s `lines` utility
// to convert our stream of bytes, `reader`, into a `Stream` of lines.
let lines = lines(BufReader::new(reader));
// Like with other small servers, we'll `spawn` this client to ensure it
// runs concurrently with all other clients. The `move` keyword is used
// here to move ownership of our db handle into the async closure.
tokio::spawn(async move {
// Since our protocol is line-based we use `tokio_codecs`'s `LineCodec`
// to convert our stream of bytes, `socket`, into a `Stream` of lines
// as well as convert our line based responses into a stream of bytes.
let mut lines = Framed::new(socket, LinesCodec::new());
// Here's where the meat of the processing in this server happens. First
// we see a clone of the database being created, which is creating a
// new reference for this connected client to use. Also note the `move`
// keyword on the closure here which moves ownership of the reference
// into the closure, which we'll need for spawning the client below.
//
// The `map` function here means that we'll run some code for all
// requests (lines) we receive from the client. The actual handling here
// is pretty simple, first we parse the request and if it's valid we
// generate a response based on the values in the database.
let db = db.clone();
let responses = lines.map(move |line| {
let request = match Request::parse(&line) {
Ok(req) => req,
Err(e) => return Response::Error { msg: e },
};
// Here for every line we get back from the `Framed` decoder,
// we parse the request, and if it's valid we generate a response
// based on the values in the database.
while let Some(result) = lines.next().await {
match result {
Ok(line) => {
let response = handle_request(&line, &db);
let mut db = db.map.lock().unwrap();
match request {
Request::Get { key } => match db.get(&key) {
Some(value) => Response::Value {
key,
value: value.clone(),
},
None => Response::Error {
msg: format!("no key {}", key),
},
},
Request::Set { key, value } => {
let previous = db.insert(key.clone(), value.clone());
Response::Set {
key,
value,
previous,
let response = response.serialize();
if let Err(e) = lines.send(response).await {
println!("error on sending response; error = {:?}", e);
}
}
Err(e) => {
println!("error on decoding from socket; error = {:?}", e);
}
}
}
}
});
// At this point `responses` is a stream of `Response` types which we
// now want to write back out to the client. To do that we use
// `Stream::fold` to perform a loop here, serializing each response and
// then writing it out to the client.
let writes = responses.fold(writer, |writer, response| {
let mut response = response.serialize();
response.push('\n');
write_all(writer, response.into_bytes()).map(|(w, _)| w)
});
// The connection will be closed at this point as `lines.next()` has returned `None`.
});
}
Err(e) => println!("error accepting socket; error = {:?}", e),
}
}
}
// Like with other small servers, we'll `spawn` this client to ensure it
// runs concurrently with all other clients, for now ignoring any errors
// that we see.
let msg = writes.then(move |_| Ok(()));
fn handle_request(line: &str, db: &Arc<Database>) -> Response {
let request = match Request::parse(&line) {
Ok(req) => req,
Err(e) => return Response::Error { msg: e },
};
tokio::spawn(msg)
});
tokio::run(done);
Ok(())
let mut db = db.map.lock().unwrap();
match request {
Request::Get { key } => match db.get(&key) {
Some(value) => Response::Value {
key,
value: value.clone(),
},
None => Response::Error {
msg: format!("no key {}", key),
},
},
Request::Set { key, value } => {
let previous = db.insert(key.clone(), value.clone());
Response::Set {
key,
value,
previous,
}
}
}
}
impl Request {
@@ -11,108 +11,82 @@
//! respectively. By default this will run I/O on all the cores your system has
//! available, and it doesn't support HTTP request bodies.
#![deny(warnings)]
extern crate bytes;
extern crate http;
extern crate httparse;
#[macro_use]
extern crate serde_derive;
extern crate serde_json;
extern crate time;
extern crate tokio;
extern crate tokio_io;
use std::net::SocketAddr;
use std::{env, fmt, io};
use tokio::codec::{Decoder, Encoder};
use tokio::net::{TcpListener, TcpStream};
use tokio::prelude::*;
#![warn(rust_2018_idioms)]
use bytes::BytesMut;
use http::header::HeaderValue;
use http::{Request, Response, StatusCode};
use futures::{SinkExt, StreamExt};
use http::{header::HeaderValue, Request, Response, StatusCode};
#[macro_use]
extern crate serde_derive;
use serde_json;
use std::{env, error::Error, fmt, io};
use tokio::net::{TcpListener, TcpStream};
use tokio_util::codec::{Decoder, Encoder, Framed};
fn main() -> Result<(), Box<std::error::Error>> {
#[tokio::main]
async fn main() -> Result<(), Box<dyn Error>> {
// Parse the arguments, bind the TCP socket we'll be listening to, spin up
// our worker threads, and start shipping sockets to those worker threads.
let addr = env::args().nth(1).unwrap_or("127.0.0.1:8080".to_string());
let addr = addr.parse::<SocketAddr>()?;
let listener = TcpListener::bind(&addr)?;
let mut server = TcpListener::bind(&addr).await?;
let mut incoming = server.incoming();
println!("Listening on: {}", addr);
tokio::run({
listener
.incoming()
.map_err(|e| println!("failed to accept socket; error = {:?}", e))
.for_each(|socket| {
process(socket);
Ok(())
})
});
while let Some(Ok(stream)) = incoming.next().await {
tokio::spawn(async move {
if let Err(e) = process(stream).await {
println!("failed to process connection; error = {}", e);
}
});
}
Ok(())
}
fn process(socket: TcpStream) {
let (tx, rx) =
// Frame the socket using the `Http` protocol. This maps the TCP socket
// to a Stream + Sink of HTTP frames.
Http.framed(socket)
// This splits a single `Stream + Sink` value into two separate handles
// that can be used independently (even on different tasks or threads).
.split();
async fn process(stream: TcpStream) -> Result<(), Box<dyn Error>> {
let mut transport = Framed::new(stream, Http);
// Map all requests into responses and send them back to the client.
let task = tx.send_all(rx.and_then(respond)).then(|res| {
if let Err(e) = res {
println!("failed to process connection; error = {:?}", e);
while let Some(request) = transport.next().await {
match request {
Ok(request) => {
let response = respond(request).await?;
transport.send(response).await?;
}
Err(e) => return Err(e.into()),
}
}
Ok(())
});
// Spawn the task that handles the connection.
tokio::spawn(task);
Ok(())
}
/// "Server logic" is implemented in this function.
///
/// This function is a map from and HTTP request to a future of a response and
/// represents the various handling a server might do. Currently the contents
/// here are pretty uninteresting.
fn respond(req: Request<()>) -> Box<Future<Item = Response<String>, Error = io::Error> + Send> {
let f = future::lazy(move || {
let mut response = Response::builder();
let body = match req.uri().path() {
"/plaintext" => {
response.header("Content-Type", "text/plain");
"Hello, World!".to_string()
}
"/json" => {
response.header("Content-Type", "application/json");
async fn respond(req: Request<()>) -> Result<Response<String>, Box<dyn Error>> {
let mut response = Response::builder();
let body = match req.uri().path() {
"/plaintext" => {
response = response.header("Content-Type", "text/plain");
"Hello, World!".to_string()
}
"/json" => {
response = response.header("Content-Type", "application/json");
#[derive(Serialize)]
struct Message {
message: &'static str,
}
serde_json::to_string(&Message {
message: "Hello, World!",
})?
#[derive(Serialize)]
struct Message {
message: &'static str,
}
_ => {
response.status(StatusCode::NOT_FOUND);
String::new()
}
};
let response = response
.body(body)
.map_err(|err| io::Error::new(io::ErrorKind::Other, err))?;
Ok(response)
});
serde_json::to_string(&Message {
message: "Hello, World!",
})?
}
_ => {
response = response.status(StatusCode::NOT_FOUND);
String::new()
}
};
let response = response
.body(body)
.map_err(|err| io::Error::new(io::ErrorKind::Other, err))?;
Box::new(f)
Ok(response)
}
struct Http;
@@ -157,13 +131,13 @@ impl Encoder for Http {
// doesn't go through io::Error.
struct BytesWrite<'a>(&'a mut BytesMut);
impl<'a> fmt::Write for BytesWrite<'a> {
impl fmt::Write for BytesWrite<'_> {
fn write_str(&mut self, s: &str) -> fmt::Result {
self.0.extend_from_slice(s.as_bytes());
Ok(())
}
fn write_fmt(&mut self, args: fmt::Arguments) -> fmt::Result {
fn write_fmt(&mut self, args: fmt::Arguments<'_>) -> fmt::Result {
fmt::write(self, args)
}
}
@@ -222,16 +196,19 @@ impl Decoder for Http {
}
let data = src.split_to(amt).freeze();
let mut ret = Request::builder();
ret.method(&data[method.0..method.1]);
ret.uri(data.slice(path.0, path.1));
ret.version(http::Version::HTTP_11);
ret = ret.method(&data[method.0..method.1]);
let s = data.slice(path.0..path.1);
let s = unsafe { String::from_utf8_unchecked(Vec::from(s.as_ref())) };
ret = ret.uri(s);
ret = ret.version(http::Version::HTTP_11);
for header in headers.iter() {
let (k, v) = match *header {
Some((ref k, ref v)) => (k, v),
None => break,
};
let value = unsafe { HeaderValue::from_shared_unchecked(data.slice(v.0, v.1)) };
ret.header(&data[k.0..k.1], value);
let value = HeaderValue::from_bytes(data.slice(v.0..v.1).as_ref())
.map_err(|_| io::Error::new(io::ErrorKind::Other, "header decode error"))?;
ret = ret.header(&data[k.0..k.1], value);
}
let req = ret
@@ -286,7 +263,7 @@ mod date {
}));
impl fmt::Display for Now {
fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
LAST.with(|cache| {
let mut cache = cache.borrow_mut();
let now = time::get_time();
@@ -313,7 +290,7 @@ mod date {
struct LocalBuffer<'a>(&'a mut LastRenderedNow);
impl<'a> fmt::Write for LocalBuffer<'a> {
impl fmt::Write for LocalBuffer<'_> {
fn write_str(&mut self, s: &str) -> fmt::Result {
let start = self.0.amt;
let end = start + s.len();
@@ -26,26 +26,27 @@
//! Please mind that since the UDP protocol doesn't have any capabilities to detect a broken
//! connection the server needs to be run first, otherwise the client will block forever.
extern crate futures;
extern crate tokio;
#![warn(rust_2018_idioms)]
use std::env;
use std::io::stdin;
use std::error::Error;
use std::io::{stdin, Read};
use std::net::SocketAddr;
use tokio::net::UdpSocket;
use tokio::prelude::*;
fn get_stdin_data() -> Result<Vec<u8>, Box<std::error::Error>> {
fn get_stdin_data() -> Result<Vec<u8>, Box<dyn std::error::Error>> {
let mut buf = Vec::new();
stdin().read_to_end(&mut buf)?;
Ok(buf)
}
fn main() -> Result<(), Box<std::error::Error>> {
#[tokio::main]
async fn main() -> Result<(), Box<dyn Error>> {
let remote_addr: SocketAddr = env::args()
.nth(1)
.unwrap_or("127.0.0.1:8080".into())
.parse()?;
// We use port 0 to let the operating system allocate an available port for us.
let local_addr: SocketAddr = if remote_addr.is_ipv4() {
"0.0.0.0:0"
@@ -53,18 +54,19 @@ fn main() -> Result<(), Box<std::error::Error>> {
"[::]:0"
}
.parse()?;
let socket = UdpSocket::bind(&local_addr)?;
let mut socket = UdpSocket::bind(local_addr).await?;
const MAX_DATAGRAM_SIZE: usize = 65_507;
socket
.send_dgram(get_stdin_data()?, &remote_addr)
.and_then(|(socket, _)| socket.recv_dgram(vec![0u8; MAX_DATAGRAM_SIZE]))
.map(|(_, data, len, _)| {
println!(
"Received {} bytes:\n{}",
len,
String::from_utf8_lossy(&data[..len])
)
})
.wait()?;
socket.connect(&remote_addr).await?;
let data = get_stdin_data()?;
socket.send(&data).await?;
let mut data = vec![0u8; MAX_DATAGRAM_SIZE];
let len = socket.recv(&mut data).await?;
println!(
"Received {} bytes:\n{}",
len,
String::from_utf8_lossy(&data[..len])
);
Ok(())
}
+77
View File
@@ -0,0 +1,77 @@
//! This example leverages `BytesCodec` to create a UDP client and server which
//! speak a custom protocol.
//!
//! Here we're using the codec from `tokio-codec` to convert a UDP socket to a stream of
//! client messages. These messages are then processed and returned back as a
//! new message with a new destination. Overall, we then use this to construct a
//! "ping pong" pair where two sockets are sending messages back and forth.
#![warn(rust_2018_idioms)]
use tokio::net::UdpSocket;
use tokio::{io, time};
use tokio_util::codec::BytesCodec;
use tokio_util::udp::UdpFramed;
use bytes::Bytes;
use futures::{FutureExt, SinkExt, StreamExt};
use std::env;
use std::error::Error;
use std::net::SocketAddr;
use std::time::Duration;
#[tokio::main]
async fn main() -> Result<(), Box<dyn Error>> {
let addr = env::args().nth(1).unwrap_or("127.0.0.1:0".to_string());
// Bind both our sockets and then figure out what ports we got.
let a = UdpSocket::bind(&addr).await?;
let b = UdpSocket::bind(&addr).await?;
let b_addr = b.local_addr()?;
let mut a = UdpFramed::new(a, BytesCodec::new());
let mut b = UdpFramed::new(b, BytesCodec::new());
// Start off by sending a ping from a to b, afterwards we just print out
// what they send us and continually send pings
let a = ping(&mut a, b_addr);
// The second client we have will receive the pings from `a` and then send
// back pongs.
let b = pong(&mut b);
// Run both futures simultaneously of `a` and `b` sending messages back and forth.
match futures::future::try_join(a, b).await {
Err(e) => println!("an error occured; error = {:?}", e),
_ => println!("done!"),
}
Ok(())
}
async fn ping(socket: &mut UdpFramed<BytesCodec>, b_addr: SocketAddr) -> Result<(), io::Error> {
socket.send((Bytes::from(&b"PING"[..]), b_addr)).await?;
for _ in 0..4usize {
let (bytes, addr) = socket.next().map(|e| e.unwrap()).await?;
println!("[a] recv: {}", String::from_utf8_lossy(&bytes));
socket.send((Bytes::from(&b"PING"[..]), addr)).await?;
}
Ok(())
}
async fn pong(socket: &mut UdpFramed<BytesCodec>) -> Result<(), io::Error> {
let timeout = Duration::from_millis(200);
while let Ok(Some(Ok((bytes, addr)))) = time::timeout(timeout, socket.next()).await {
println!("[b] recv: {}", String::from_utf8_lossy(&bytes));
socket.send((Bytes::from(&b"PONG"[..]), addr)).await?;
}
Ok(())
}
+1
View File
@@ -0,0 +1 @@
edition = "2018"
+15
View File
@@ -0,0 +1,15 @@
[package]
name = "tests-build"
version = "0.1.0"
authors = ["Tokio Contributors <[email protected]>"]
edition = "2018"
publish = false
[features]
full = ["tokio/full"]
[dependencies]
tokio = { path = "../tokio", optional = true }
[dev-dependencies]
trybuild = "1.0"
+2
View File
@@ -0,0 +1,2 @@
Tests the various combination of feature flags. This is broken out to a separate
crate to work around limitations with cargo features.
+2
View File
@@ -0,0 +1,2 @@
#[cfg(feature = "tokio")]
pub use tokio;
@@ -0,0 +1,25 @@
use tests_build::tokio;
#[tokio::main]
fn main_is_not_async() {}
#[tokio::main(foo)]
async fn main_attr_has_unknown_args() {}
#[tokio::main(threadpool::bar)]
async fn main_attr_has_path_args() {}
#[tokio::test]
fn test_is_not_async() {}
#[tokio::test]
async fn test_fn_has_args(_x: u8) {}
#[tokio::test(foo)]
async fn test_attr_has_args() {}
#[tokio::test]
#[test]
async fn test_has_second_test_attr() {}
fn main() {}
@@ -0,0 +1,41 @@
error: the async keyword is missing from the function declaration
--> $DIR/macros_invalid_input.rs:4:1
|
4 | fn main_is_not_async() {}
| ^^
error: Unknown attribute foo is specified; expected `basic_scheduler` or `threaded_scheduler`
--> $DIR/macros_invalid_input.rs:6:15
|
6 | #[tokio::main(foo)]
| ^^^
error: Must have specified ident
--> $DIR/macros_invalid_input.rs:9:15
|
9 | #[tokio::main(threadpool::bar)]
| ^^^^^^^^^^^^^^^
error: the async keyword is missing from the function declaration
--> $DIR/macros_invalid_input.rs:13:1
|
13 | fn test_is_not_async() {}
| ^^
error: the test function cannot accept arguments
--> $DIR/macros_invalid_input.rs:16:27
|
16 | async fn test_fn_has_args(_x: u8) {}
| ^^^^^^
error: Unknown attribute foo is specified; expected `basic_scheduler` or `threaded_scheduler`
--> $DIR/macros_invalid_input.rs:18:15
|
18 | #[tokio::test(foo)]
| ^^^
error: second test attribute is supplied
--> $DIR/macros_invalid_input.rs:22:1
|
22 | #[test]
| ^^^^^^^
+9
View File
@@ -0,0 +1,9 @@
#[test]
fn compile_fail() {
let t = trybuild::TestCases::new();
#[cfg(feature = "full")]
t.compile_fail("tests/fail/macros_invalid_input.rs");
drop(t);
}
+15
View File
@@ -0,0 +1,15 @@
[package]
name = "tests-integration"
version = "0.1.0"
authors = ["Tokio Contributors <[email protected]>"]
edition = "2018"
publish = false
[dependencies]
tokio = { path = "../tokio", features = ["full"] }
doc-comment = "0.3.1"
[dev-dependencies]
tokio-test = { path = "../tokio-test" }
futures = { version = "0.3.0", features = ["async-await"] }
+1
View File
@@ -0,0 +1 @@
Tests that require additional components than just the `tokio` crate.
+20
View File
@@ -0,0 +1,20 @@
//! A cat-like utility that can be used as a subprocess to test I/O
//! stream communication.
use std::io;
use std::io::Write;
fn main() {
let stdin = io::stdin();
let mut stdout = io::stdout();
let mut line = String::new();
loop {
line.clear();
stdin.read_line(&mut line).unwrap();
if line.is_empty() {
break;
}
stdout.write_all(line.as_bytes()).unwrap();
}
stdout.flush().unwrap();
}
+4
View File
@@ -0,0 +1,4 @@
use doc_comment::doc_comment;
// #[doc = include_str!("../../README.md")]
doc_comment!(include_str!("../../README.md"));
+126
View File
@@ -0,0 +1,126 @@
#![warn(rust_2018_idioms)]
use tokio::io::{AsyncBufReadExt, AsyncWriteExt, BufReader};
use tokio::process::{Child, Command};
use tokio_test::assert_ok;
use futures::future::{self, FutureExt};
use std::env;
use std::io;
use std::process::{ExitStatus, Stdio};
fn cat() -> Command {
let mut me = env::current_exe().unwrap();
me.pop();
if me.ends_with("deps") {
me.pop();
}
me.push("test-cat");
let mut cmd = Command::new(me);
cmd.stdin(Stdio::piped()).stdout(Stdio::piped());
cmd
}
async fn feed_cat(mut cat: Child, n: usize) -> io::Result<ExitStatus> {
let mut stdin = cat.stdin().take().unwrap();
let stdout = cat.stdout().take().unwrap();
// Produce n lines on the child's stdout.
let write = async {
for i in 0..n {
let bytes = format!("line {}\n", i).into_bytes();
stdin.write_all(&bytes).await.unwrap();
}
drop(stdin);
};
let read = async {
let mut reader = BufReader::new(stdout).lines();
let mut num_lines = 0;
// Try to read `n + 1` lines, ensuring the last one is empty
// (i.e. EOF is reached after `n` lines.
loop {
let data = reader
.next_line()
.await
.unwrap_or_else(|_| Some(String::new()))
.expect("failed to read line");
let num_read = data.len();
let done = num_lines >= n;
match (done, num_read) {
(false, 0) => panic!("broken pipe"),
(true, n) if n != 0 => panic!("extraneous data"),
_ => {
let expected = format!("line {}", num_lines);
assert_eq!(expected, data);
}
};
num_lines += 1;
if num_lines >= n {
break;
}
}
};
// Compose reading and writing concurrently.
future::join3(write, read, cat)
.map(|(_, _, status)| status)
.await
}
/// Check for the following properties when feeding stdin and
/// consuming stdout of a cat-like process:
///
/// - A number of lines that amounts to a number of bytes exceeding a
/// typical OS buffer size can be fed to the child without
/// deadlock. This tests that we also consume the stdout
/// concurrently; otherwise this would deadlock.
///
/// - We read the same lines from the child that we fed it.
///
/// - The child does produce EOF on stdout after the last line.
#[tokio::test]
async fn feed_a_lot() {
let child = cat().spawn().unwrap();
let status = feed_cat(child, 10000).await.unwrap();
assert_eq!(status.code(), Some(0));
}
#[tokio::test]
async fn wait_with_output_captures() {
let mut child = cat().spawn().unwrap();
let mut stdin = child.stdin().take().unwrap();
let write_bytes = b"1234";
let future = async {
stdin.write_all(write_bytes).await?;
drop(stdin);
let out = child.wait_with_output();
out.await
};
let output = future.await.unwrap();
assert!(output.status.success());
assert_eq!(output.stdout, write_bytes);
assert_eq!(output.stderr.len(), 0);
}
#[tokio::test]
async fn status_closes_any_pipes() {
// Cat will open a pipe between the parent and child.
// If `status_async` doesn't ensure the handles are closed,
// we would end up blocking forever (and time out).
let child = cat().status();
assert_ok!(child.await);
}
-14
View File
@@ -1,14 +0,0 @@
# 0.1.1 (April 22, 2019)
### Added
- Utilities for creating a `BufStream` from iterators and streams (#1011).
- Add `BufStream::into_stream` (#1048).
- Implement `FromBufStream` for `Bytes` (#1009).
- Implement `Error` for `CollectVecError` (#1010).
### Fixed
- Implement `size_hint` for string types (#1012).
# 0.1.0 (February 23, 2019)
* Initial release
-32
View File
@@ -1,32 +0,0 @@
[package]
name = "tokio-buf"
# When releasing to crates.io:
# - Remove path dependencies
# - Update html_root_url.
# - Update doc url
# - Cargo.toml
# - README.md
# - Update CHANGELOG.md.
# - Create "v0.1.x" git tag.
version = "0.1.1"
authors = ["Carl Lerche <[email protected]>"]
license = "MIT"
repository = "https://github.com/tokio-rs/tokio"
homepage = "https://tokio.rs"
documentation = "https://docs.rs/tokio-buf/0.1.1/tokio_buf"
description = """
Asynchronous stream of byte buffers
"""
categories = ["asynchronous"]
[dependencies]
bytes = "0.4.10"
either = { version = "1.5", optional = true}
futures = "0.1.23"
[features]
default = ["util"]
util = ["bytes/either", "either"]
[dev-dependencies]
tokio-mock-task = "0.1.1"
-35
View File
@@ -1,35 +0,0 @@
# tokio-buf
Asynchronous stream of byte buffers
[Documenation](https://docs.rs/tokio-buf)
## Usage
First, add this to your `Cargo.toml`:
```toml
[dependencies]
tokio-buf = "0.1.1"
```
Next, add this to your crate:
```rust
extern crate tokio_buf;
```
You can find extensive documentation and examples about how to use this crate
online at [https://tokio.rs](https://tokio.rs). The [API
documentation](https://docs.rs/tokio-buf) is also a great place to get started
for the nitty-gritty.
## License
This project is licensed under the [MIT license](LICENSE).
### Contribution
Unless you explicitly state otherwise, any contribution intentionally submitted
for inclusion in Tokio by you, shall be licensed as MIT, without any additional
terms or conditions.
-99
View File
@@ -1,99 +0,0 @@
#![doc(html_root_url = "https://docs.rs/tokio-buf/0.1.1")]
#![deny(missing_docs, missing_debug_implementations, unreachable_pub)]
#![cfg_attr(test, deny(warnings))]
//! Asynchronous stream of bytes.
//!
//! This crate contains the `BufStream` trait and a number of combinators for
//! this trait. The trait is similar to `Stream` in the `futures` library, but
//! instead of yielding arbitrary values, it only yields types that implement
//! `Buf` (i.e, byte collections).
extern crate bytes;
#[cfg(feature = "util")]
extern crate either;
#[allow(unused)]
#[macro_use]
extern crate futures;
mod never;
mod size_hint;
mod str;
mod u8;
#[cfg(feature = "util")]
pub mod util;
pub use self::size_hint::SizeHint;
#[doc(inline)]
#[cfg(feature = "util")]
pub use util::BufStreamExt;
use bytes::Buf;
use futures::Poll;
/// An asynchronous stream of bytes.
///
/// `BufStream` asynchronously yields values implementing `Buf`, i.e. byte
/// buffers.
pub trait BufStream {
/// Values yielded by the `BufStream`.
///
/// Each item is a sequence of bytes representing a chunk of the total
/// `ByteStream`.
type Item: Buf;
/// The error type this `BufStream` might generate.
type Error;
/// Attempt to pull out the next buffer of this stream, registering the
/// current task for wakeup if the value is not yet available, and returning
/// `None` if the stream is exhausted.
///
/// # Return value
///
/// There are several possible return values, each indicating a distinct
/// stream state:
///
/// - `Ok(Async::NotReady)` means that this stream's next value is not ready
/// yet. Implementations will ensure that the current task will be notified
/// when the next value may be ready.
///
/// - `Ok(Async::Ready(Some(buf)))` means that the stream has successfully
/// produced a value, `buf`, and may produce further values on subsequent
/// `poll_buf` calls.
///
/// - `Ok(Async::Ready(None))` means that the stream has terminated, and
/// `poll_buf` should not be invoked again.
///
/// # Panics
///
/// Once a stream is finished, i.e. `Ready(None)` has been returned, further
/// calls to `poll_buf` may result in a panic or other "bad behavior".
fn poll_buf(&mut self) -> Poll<Option<Self::Item>, Self::Error>;
/// Returns the bounds on the remaining length of the stream.
///
/// The size hint allows the caller to perform certain optimizations that
/// are dependent on the byte stream size. For example, `collect` uses the
/// size hint to pre-allocate enough capacity to store the entirety of the
/// data received from the byte stream.
///
/// When `SizeHint::upper()` returns `Some` with a value equal to
/// `SizeHint::lower()`, this represents the exact number of bytes that will
/// be yielded by the `BufStream`.
///
/// # Implementation notes
///
/// While not enforced, implementations are expected to respect the values
/// returned from `SizeHint`. Any deviation is considered an implementation
/// bug. Consumers may rely on correctness in order to use the value as part
/// of protocol impelmentations. For example, an HTTP library may use the
/// size hint to set the `content-length` header.
///
/// However, `size_hint` must not be trusted to omit bounds checks in unsafe
/// code. An incorrect implementation of `size_hint()` must not lead to
/// memory safety violations.
fn size_hint(&self) -> SizeHint {
SizeHint::default()
}
}
-22
View File
@@ -1,22 +0,0 @@
use std::{error, fmt};
/// An error that can never occur
pub enum Never {}
impl fmt::Debug for Never {
fn fmt(&self, _f: &mut fmt::Formatter) -> fmt::Result {
match *self {}
}
}
impl fmt::Display for Never {
fn fmt(&self, _f: &mut fmt::Formatter) -> fmt::Result {
match *self {}
}
}
impl error::Error for Never {
fn description(&self) -> &str {
match *self {}
}
}
-56
View File
@@ -1,56 +0,0 @@
use std::u64;
/// A `BufStream` size hint
///
/// The default implementation returns:
///
/// * 0 for `available`
/// * 0 for `lower`
/// * `None` for `upper`.
#[derive(Debug, Default, Clone)]
pub struct SizeHint {
lower: u64,
upper: Option<u64>,
}
impl SizeHint {
/// Returns a new `SizeHint` with default values
pub fn new() -> SizeHint {
SizeHint::default()
}
/// Returns the lower bound of data that the `BufStream` will yield before
/// completing.
pub fn lower(&self) -> u64 {
self.lower
}
/// Set the value of the `lower` hint.
///
/// # Panics
///
/// The function panics if `value` is greater than `upper`.
pub fn set_lower(&mut self, value: u64) {
assert!(value <= self.upper.unwrap_or(u64::MAX));
self.lower = value;
}
/// Returns the upper bound of data the `BufStream` will yield before
/// completing, or `None` if the value is unknown.
pub fn upper(&self) -> Option<u64> {
self.upper
}
/// Set the value of the `upper` hint value.
///
/// # Panics
///
/// This function panics if `value` is less than `lower`.
pub fn set_upper(&mut self, value: u64) {
// There is no need to check `available` as that is guaranteed to be
// less than or equal to `lower`.
assert!(value >= self.lower, "`value` is less than than `lower`");
self.upper = Some(value);
}
}
-55
View File
@@ -1,55 +0,0 @@
use never::Never;
use BufStream;
use SizeHint;
use futures::Poll;
use std::io;
use std::mem;
impl BufStream for String {
type Item = io::Cursor<Vec<u8>>;
type Error = Never;
fn poll_buf(&mut self) -> Poll<Option<Self::Item>, Self::Error> {
if self.is_empty() {
return Ok(None.into());
}
let bytes = mem::replace(self, Default::default()).into_bytes();
let buf = io::Cursor::new(bytes);
Ok(Some(buf).into())
}
fn size_hint(&self) -> SizeHint {
size_hint(&self[..])
}
}
impl BufStream for &'static str {
type Item = io::Cursor<&'static [u8]>;
type Error = Never;
fn poll_buf(&mut self) -> Poll<Option<Self::Item>, Self::Error> {
if self.is_empty() {
return Ok(None.into());
}
let bytes = mem::replace(self, Default::default()).as_bytes();
let buf = io::Cursor::new(bytes);
Ok(Some(buf).into())
}
fn size_hint(&self) -> SizeHint {
size_hint(&self[..])
}
}
fn size_hint(s: &str) -> SizeHint {
let mut hint = SizeHint::new();
hint.set_lower(s.len() as u64);
hint.set_upper(s.len() as u64);
hint
}
-66
View File
@@ -1,66 +0,0 @@
use bytes::{Bytes, BytesMut};
use futures::Poll;
use never::Never;
use std::io;
use BufStream;
impl BufStream for Vec<u8> {
type Item = io::Cursor<Vec<u8>>;
type Error = Never;
fn poll_buf(&mut self) -> Poll<Option<Self::Item>, Self::Error> {
if self.is_empty() {
return Ok(None.into());
}
poll_bytes(self)
}
}
impl BufStream for &'static [u8] {
type Item = io::Cursor<&'static [u8]>;
type Error = Never;
fn poll_buf(&mut self) -> Poll<Option<Self::Item>, Self::Error> {
if self.is_empty() {
return Ok(None.into());
}
poll_bytes(self)
}
}
impl BufStream for Bytes {
type Item = io::Cursor<Bytes>;
type Error = Never;
fn poll_buf(&mut self) -> Poll<Option<Self::Item>, Self::Error> {
if self.is_empty() {
return Ok(None.into());
}
poll_bytes(self)
}
}
impl BufStream for BytesMut {
type Item = io::Cursor<BytesMut>;
type Error = Never;
fn poll_buf(&mut self) -> Poll<Option<Self::Item>, Self::Error> {
if self.is_empty() {
return Ok(None.into());
}
poll_bytes(self)
}
}
fn poll_bytes<T: Default>(buf: &mut T) -> Poll<Option<io::Cursor<T>>, Never> {
use std::mem;
let bytes = mem::replace(buf, Default::default());
let buf = io::Cursor::new(bytes);
Ok(Some(buf).into())
}
-46
View File
@@ -1,46 +0,0 @@
use BufStream;
use either::Either;
use futures::Poll;
/// A buf stream that sequences two buf streams together.
///
/// `Chain` values are produced by the `chain` function on `BufStream`.
#[derive(Debug)]
pub struct Chain<T, U> {
left: Option<T>,
right: U,
}
impl<T, U> Chain<T, U> {
pub(crate) fn new(left: T, right: U) -> Chain<T, U> {
Chain {
left: Some(left),
right,
}
}
}
impl<T, U> BufStream for Chain<T, U>
where
T: BufStream,
U: BufStream<Error = T::Error>,
{
type Item = Either<T::Item, U::Item>;
type Error = T::Error;
fn poll_buf(&mut self) -> Poll<Option<Self::Item>, Self::Error> {
if let Some(ref mut stream) = self.left {
let res = try_ready!(stream.poll_buf());
if res.is_some() {
return Ok(res.map(Either::Left).into());
}
}
self.left = None;
let res = try_ready!(self.right.poll_buf());
Ok(res.map(Either::Right).into())
}
}
-101
View File
@@ -1,101 +0,0 @@
use super::FromBufStream;
use BufStream;
use futures::{Future, Poll};
/// Consumes a buf stream, collecting the data into a single byte container.
///
/// `Collect` values are produced by `BufStream::collect`.
#[derive(Debug)]
pub struct Collect<T, U>
where
T: BufStream,
U: FromBufStream<T::Item>,
{
stream: T,
builder: Option<U::Builder>,
}
/// Errors returned from `Collect` future.
#[derive(Debug)]
pub struct CollectError<T, U> {
inner: Error<T, U>,
}
#[derive(Debug)]
enum Error<T, U> {
Stream(T),
Collect(U),
}
impl<T, U> Collect<T, U>
where
T: BufStream,
U: FromBufStream<T::Item>,
{
pub(crate) fn new(stream: T) -> Collect<T, U> {
let builder = U::builder(&stream.size_hint());
Collect {
stream,
builder: Some(builder),
}
}
}
impl<T, U> Future for Collect<T, U>
where
T: BufStream,
U: FromBufStream<T::Item>,
{
type Item = U;
type Error = CollectError<T::Error, U::Error>;
fn poll(&mut self) -> Poll<Self::Item, Self::Error> {
loop {
let res = self.stream.poll_buf().map_err(|err| {
let inner = Error::Stream(err);
CollectError { inner }
});
match try_ready!(res) {
Some(mut buf) => {
let builder = self.builder.as_mut().expect("cannot poll after done");
U::extend(builder, &mut buf, &self.stream.size_hint()).map_err(|err| {
let inner = Error::Collect(err);
CollectError { inner }
})?;
}
None => {
let builder = self.builder.take().expect("cannot poll after done");
let value = U::build(builder).map_err(|err| {
let inner = Error::Collect(err);
CollectError { inner }
})?;
return Ok(value.into());
}
}
}
}
}
// ===== impl CollectError =====
impl<T, U> CollectError<T, U> {
/// Returns `true` if the error was caused by polling the stream.
pub fn is_stream_err(&self) -> bool {
match self.inner {
Error::Stream(_) => true,
_ => false,
}
}
/// Returns `true` if the error happened while collecting the data.
pub fn is_collect_err(&self) -> bool {
match self.inner {
Error::Collect(_) => true,
_ => false,
}
}
}
-162
View File
@@ -1,162 +0,0 @@
use SizeHint;
use bytes::{Buf, BufMut, Bytes};
use std::error::Error;
use std::fmt;
use std::usize;
/// Conversion from a `BufStream`.
///
/// By implementing `FromBufStream` for a type, you define how it will be
/// created from a buf stream. This is common for types which describe byte
/// storage of some kind.
///
/// `FromBufStream` is rarely called explicitly, and it is instead used through
/// `BufStream`'s `collect` method.
pub trait FromBufStream<T: Buf>: Sized {
/// Type that is used to build `Self` while the `BufStream` is being
/// consumed.
type Builder;
/// Error that might happen on conversion.
type Error;
/// Create a new, empty, builder. The provided `hint` can be used to inform
/// reserving capacity.
fn builder(hint: &SizeHint) -> Self::Builder;
/// Extend the builder with the `Buf`.
///
/// This method is called whenever a new `Buf` value is obtained from the
/// buf stream.
///
/// The provided size hint represents the state of the stream **after**
/// `buf` has been yielded. The lower bound represents the minimum amount of
/// data that will be provided after this call to `extend` returns.
fn extend(builder: &mut Self::Builder, buf: &mut T, hint: &SizeHint)
-> Result<(), Self::Error>;
/// Finalize the building of `Self`.
///
/// Called once the buf stream is fully consumed.
fn build(builder: Self::Builder) -> Result<Self, Self::Error>;
}
/// Error returned from collecting into a `Vec<u8>`
#[derive(Debug)]
pub struct CollectVecError {
_p: (),
}
/// Error returned from collecting into a `Bytes`
#[derive(Debug)]
pub struct CollectBytesError {
_p: (),
}
impl<T: Buf> FromBufStream<T> for Vec<u8> {
type Builder = Vec<u8>;
type Error = CollectVecError;
fn builder(hint: &SizeHint) -> Vec<u8> {
Vec::with_capacity(hint.lower() as usize)
}
fn extend(builder: &mut Self, buf: &mut T, hint: &SizeHint) -> Result<(), Self::Error> {
let lower = hint.lower();
// If the lower bound is greater than `usize::MAX` then we have a
// problem
if lower > usize::MAX as u64 {
return Err(CollectVecError { _p: () });
}
let mut reserve = lower as usize;
// If `upper` is set, use this value if it is less than or equal to 64.
// This only really impacts the first iteration.
match hint.upper() {
Some(upper) if upper <= 64 => {
reserve = upper as usize;
}
_ => {}
}
// hint.lower() represents the minimum amount of data that will be
// received *after* this function call. We reserve this amount on top of
// the amount of data in `buf`.
reserve = match reserve.checked_add(buf.remaining()) {
Some(n) => n,
None => return Err(CollectVecError { _p: () }),
};
// Always reserve 64 bytes the first time, unless `upper` is set and is
// less than 64.
if builder.is_empty() {
reserve = reserve.max(match hint.upper() {
Some(upper) if upper < 64 => upper as usize,
_ => 64,
});
}
// Make sure overflow won't happen when reserving
if reserve.checked_add(builder.len()).is_none() {
return Err(CollectVecError { _p: () });
}
// Reserve space
builder.reserve(reserve);
// Copy the data
builder.put(buf);
Ok(())
}
fn build(builder: Self) -> Result<Self, Self::Error> {
Ok(builder)
}
}
impl<T: Buf> FromBufStream<T> for Bytes {
type Builder = Vec<u8>;
type Error = CollectBytesError;
fn builder(hint: &SizeHint) -> Vec<u8> {
<Vec<u8> as FromBufStream<T>>::builder(hint)
}
fn extend(builder: &mut Vec<u8>, buf: &mut T, hint: &SizeHint) -> Result<(), Self::Error> {
<Vec<u8> as FromBufStream<T>>::extend(builder, buf, hint)
.map_err(|_| CollectBytesError { _p: () })
}
fn build(builder: Vec<u8>) -> Result<Self, Self::Error> {
Ok(builder.into())
}
}
impl fmt::Display for CollectVecError {
fn fmt(&self, fmt: &mut fmt::Formatter) -> fmt::Result {
write!(fmt, "BufStream is too big")
}
}
impl Error for CollectVecError {
fn description(&self) -> &str {
"BufStream too big"
}
}
impl fmt::Display for CollectBytesError {
fn fmt(&self, fmt: &mut fmt::Formatter) -> fmt::Result {
write!(fmt, "BufStream too big")
}
}
impl Error for CollectBytesError {
fn description(&self) -> &str {
"BufStream too big"
}
}
-54
View File
@@ -1,54 +0,0 @@
use bytes::Buf;
use futures::Poll;
use std::error::Error;
use std::fmt;
use BufStream;
/// Converts an `Iterator` into a `BufStream` which is always ready to yield the
/// next value.
///
/// Iterators in Rust don't express the ability to block, so this adapter
/// simply always calls `iter.next()` and returns that.
pub fn iter<I>(i: I) -> Iter<I::IntoIter>
where
I: IntoIterator,
I::Item: Buf,
{
Iter {
iter: i.into_iter(),
}
}
/// `BufStream` returned by the [`iter`] function.
#[derive(Debug)]
pub struct Iter<I> {
iter: I,
}
#[derive(Debug)]
pub enum Never {}
impl<I> BufStream for Iter<I>
where
I: Iterator,
I::Item: Buf,
{
type Item = I::Item;
type Error = Never;
fn poll_buf(&mut self) -> Poll<Option<Self::Item>, Self::Error> {
Ok(self.iter.next().into())
}
}
impl fmt::Display for Never {
fn fmt(&self, _: &mut fmt::Formatter) -> fmt::Result {
unreachable!();
}
}
impl Error for Never {
fn description(&self) -> &str {
unreachable!();
}
}
-76
View File
@@ -1,76 +0,0 @@
use BufStream;
use bytes::Buf;
use futures::Poll;
/// Limits the stream to a maximum amount of data.
#[derive(Debug)]
pub struct Limit<T> {
stream: T,
remaining: u64,
}
/// Errors returned from `Limit`.
#[derive(Debug)]
pub struct LimitError<T> {
/// When `None`, limit was reached
inner: Option<T>,
}
impl<T> Limit<T> {
pub(crate) fn new(stream: T, amount: u64) -> Limit<T> {
Limit {
stream,
remaining: amount,
}
}
}
impl<T> BufStream for Limit<T>
where
T: BufStream,
{
type Item = T::Item;
type Error = LimitError<T::Error>;
fn poll_buf(&mut self) -> Poll<Option<Self::Item>, Self::Error> {
use futures::Async::Ready;
if self.stream.size_hint().lower() > self.remaining {
return Err(LimitError { inner: None });
}
let res = self
.stream
.poll_buf()
.map_err(|err| LimitError { inner: Some(err) });
match res {
Ok(Ready(Some(ref buf))) => {
if buf.remaining() as u64 > self.remaining {
self.remaining = 0;
return Err(LimitError { inner: None });
}
self.remaining -= buf.remaining() as u64;
}
_ => {}
}
res
}
}
// ===== impl LimitError =====
impl<T> LimitError<T> {
/// Returns `true` if the error was caused by polling the stream.
pub fn is_stream_err(&self) -> bool {
self.inner.is_some()
}
/// Returns `true` if the stream reached its limit.
pub fn is_limit_err(&self) -> bool {
self.inner.is_none()
}
}
-87
View File
@@ -1,87 +0,0 @@
//! Types and utilities for working with `BufStream`.
mod chain;
mod collect;
mod from;
mod iter;
mod limit;
mod stream;
pub use self::chain::Chain;
pub use self::collect::Collect;
pub use self::from::FromBufStream;
pub use self::iter::iter;
pub use self::limit::Limit;
pub use self::stream::{stream, IntoStream};
pub mod error {
//! Error types
pub use super::collect::CollectError;
pub use super::from::{CollectBytesError, CollectVecError};
pub use super::limit::LimitError;
}
use BufStream;
impl<T> BufStreamExt for T where T: BufStream {}
/// An extension trait for `BufStream`'s that provides a variety of convenient
/// adapters.
pub trait BufStreamExt: BufStream {
/// Takes two buf streams and creates a new buf stream over both in
/// sequence.
///
/// `chain()` returns a new `BufStream` value which will first yield all
/// data from `self` then all data from `other`.
///
/// In other words, it links two buf streams together, in a chain.
fn chain<T>(self, other: T) -> Chain<Self, T>
where
Self: Sized,
T: BufStream<Error = Self::Error>,
{
Chain::new(self, other)
}
/// Consumes all data from `self`, storing it in byte storage of type `T`.
///
/// `collect()` returns a future that buffers all data yielded from `self`
/// into storage of type of `T`. The future completes once `self` yield
/// `None`, returning the buffered data.
///
/// The collect future will yield an error if `self` yields an error or if
/// the collect operation errors. The collect error cases are dependent on
/// the target storage type.
fn collect<T>(self) -> Collect<Self, T>
where
Self: Sized,
T: FromBufStream<Self::Item>,
{
Collect::new(self)
}
/// Limit the number of bytes that the stream can yield.
///
/// `limit()` returns a new `BufStream` value which yields all the data from
/// `self` while ensuring that at most `amount` bytes are yielded.
///
/// If `self` can yield greater than `amount` bytes, the returned stream
/// will yield an error.
fn limit(self, amount: u64) -> Limit<Self>
where
Self: Sized,
{
Limit::new(self, amount)
}
/// Creates a `Stream` from a `BufStream`.
///
/// This produces a `Stream` of `BufStream::Items`.
fn into_stream(self) -> IntoStream<Self>
where
Self: Sized,
{
IntoStream::new(self)
}
}
-76
View File
@@ -1,76 +0,0 @@
use bytes::Buf;
use futures::{Async, Poll, Stream};
use BufStream;
/// Converts a `Stream` of `Buf` types into a `BufStream`.
///
/// While `Stream` and `BufStream` are very similar, they are not identical. The
/// `stream` function returns a `BufStream` that is backed by the provided
/// `Stream` type.
pub fn stream<T>(stream: T) -> FromStream<T>
where
T: Stream,
T::Item: Buf,
{
FromStream { stream }
}
/// `BufStream` returned by the [`stream`] function.
#[derive(Debug)]
pub struct FromStream<T> {
stream: T,
}
impl<T> BufStream for FromStream<T>
where
T: Stream,
T::Item: Buf,
{
type Item = T::Item;
type Error = T::Error;
fn poll_buf(&mut self) -> Poll<Option<Self::Item>, Self::Error> {
self.stream.poll()
}
}
/// Converts a `BufStream` into a `Stream`.
#[derive(Debug)]
pub struct IntoStream<T> {
buf: T,
}
impl<T> IntoStream<T> {
/// Create a new `Stream` from the provided `BufStream`.
pub fn new(buf: T) -> Self {
IntoStream { buf }
}
/// Get a reference to the inner `BufStream`.
pub fn get_ref(&self) -> &T {
&self.buf
}
/// Get a mutable reference to the inner `BufStream`
pub fn get_mut(&mut self) -> &mut T {
&mut self.buf
}
/// Get the inner `BufStream`.
pub fn into_inner(self) -> T {
self.buf
}
}
impl<T: BufStream> Stream for IntoStream<T> {
type Item = T::Item;
type Error = T::Error;
fn poll(&mut self) -> Poll<Option<Self::Item>, Self::Error> {
match self.buf.poll_buf()? {
Async::Ready(Some(buf)) => Ok(Async::Ready(Some(buf))),
Async::Ready(None) => Ok(Async::Ready(None)),
Async::NotReady => Ok(Async::NotReady),
}
}
}
-7
View File
@@ -1,7 +0,0 @@
extern crate tokio_buf;
use tokio_buf::BufStream;
// Ensures that `BufStream` can be a trait object
#[allow(dead_code)]
fn obj(_: &mut BufStream<Item = u32, Error = ()>) {}
-43
View File
@@ -1,43 +0,0 @@
#![cfg(feature = "util")]
extern crate bytes;
extern crate futures;
extern crate tokio_buf;
use futures::Async::*;
use tokio_buf::{BufStream, BufStreamExt};
#[macro_use]
mod support;
use support::*;
#[test]
fn chain() {
// Chain one with one
//
let mut bs = one("hello").chain(one("world"));
assert_buf_eq!(bs.poll_buf(), "hello");
assert_buf_eq!(bs.poll_buf(), "world");
assert_none!(bs.poll_buf());
// Chain multi with multi
let mut bs = list(&["foo", "bar"]).chain(list(&["baz", "bok"]));
assert_buf_eq!(bs.poll_buf(), "foo");
assert_buf_eq!(bs.poll_buf(), "bar");
assert_buf_eq!(bs.poll_buf(), "baz");
assert_buf_eq!(bs.poll_buf(), "bok");
assert_none!(bs.poll_buf());
// Chain includes a not ready call
//
let mut bs = new_mock(&[Ok(Ready("foo")), Ok(NotReady), Ok(Ready("bar"))]).chain(one("baz"));
assert_buf_eq!(bs.poll_buf(), "foo");
assert_not_ready!(bs.poll_buf());
assert_buf_eq!(bs.poll_buf(), "bar");
assert_buf_eq!(bs.poll_buf(), "baz");
assert_none!(bs.poll_buf());
}
-68
View File
@@ -1,68 +0,0 @@
#![cfg(feature = "util")]
extern crate bytes;
extern crate futures;
extern crate tokio_buf;
use bytes::Bytes;
use futures::Future;
use tokio_buf::BufStreamExt;
#[macro_use]
mod support;
use support::*;
macro_rules! test_collect_impl {
($t:ty $(, $capacity:ident)*) => {
// While unfortunate, this test makes some assumptions on vec's resizing
// behavior.
//
// Collect one
//
let bs = one("hello world");
let vec: $t = bs.collect().wait().unwrap();
assert_eq!(vec, &b"hello world"[..]);
$( assert_eq!(vec.$capacity(), 64); )*
// Collect one, with size hint
//
let mut bs = one("hello world");
bs.size_hint.set_lower(11);
let vec: $t = bs.collect().wait().unwrap();
assert_eq!(vec, &b"hello world"[..]);
$( assert_eq!(vec.$capacity(), 64); )*
// Collect one, with size hint
//
let mut bs = one("hello world");
bs.size_hint.set_lower(10);
let vec: $t = bs.collect().wait().unwrap();
assert_eq!(vec, &b"hello world"[..]);
$( assert_eq!(vec.$capacity(), 64); )*
// Collect many
//
let bs = list(&["hello", " ", "world", ", one two three"]);
let vec: $t = bs.collect().wait().unwrap();
assert_eq!(vec, &b"hello world, one two three"[..]);
}
}
#[test]
fn collect_vec() {
test_collect_impl!(Vec<u8>, capacity);
}
#[test]
fn collect_bytes() {
test_collect_impl!(Bytes);
}
-33
View File
@@ -1,33 +0,0 @@
extern crate bytes;
extern crate futures;
extern crate tokio_buf;
use futures::Async::*;
use std::io::Cursor;
use tokio_buf::{util, BufStream};
#[macro_use]
mod support;
type Buf = Cursor<&'static [u8]>;
#[test]
fn empty_iter() {
let mut bs = util::iter(Vec::<Buf>::new());
assert_none!(bs.poll_buf());
}
#[test]
fn full_iter() {
let bufs = vec![buf(b"one"), buf(b"two"), buf(b"three")];
let mut bs = util::iter(bufs);
assert_buf_eq!(bs.poll_buf(), "one");
assert_buf_eq!(bs.poll_buf(), "two");
assert_buf_eq!(bs.poll_buf(), "three");
assert_none!(bs.poll_buf());
}
fn buf(data: &'static [u8]) -> Buf {
Cursor::new(data)
}
-65
View File
@@ -1,65 +0,0 @@
#![cfg(feature = "util")]
extern crate bytes;
extern crate futures;
extern crate tokio_buf;
use futures::Async::*;
use futures::Future;
use tokio_buf::{BufStream, BufStreamExt};
#[macro_use]
mod support;
use support::*;
#[test]
fn limit() {
// Not limited
let res = one("hello world")
.limit(100)
.collect::<Vec<_>>()
.wait()
.unwrap();
assert_eq!(res, b"hello world");
let res = list(&["hello", " ", "world"])
.limit(100)
.collect::<Vec<_>>()
.wait()
.unwrap();
assert_eq!(res, b"hello world");
let res = list(&["hello", " ", "world"])
.limit(11)
.collect::<Vec<_>>()
.wait()
.unwrap();
assert_eq!(res, b"hello world");
// Limited
let res = one("hello world").limit(5).collect::<Vec<_>>().wait();
assert!(res.is_err());
let res = one("hello world").limit(10).collect::<Vec<_>>().wait();
assert!(res.is_err());
let mut bs = list(&["hello", " ", "world"]).limit(9);
assert_buf_eq!(bs.poll_buf(), "hello");
assert_buf_eq!(bs.poll_buf(), " ");
assert!(bs.poll_buf().is_err());
let mut bs = list(&["hello", " ", "world"]);
bs.size_hint.set_lower(11);
let mut bs = bs.limit(9);
assert!(bs.poll_buf().is_err());
}
-42
View File
@@ -1,42 +0,0 @@
extern crate tokio_buf;
use tokio_buf::SizeHint;
#[test]
fn size_hint() {
let hint = SizeHint::new();
assert_eq!(hint.lower(), 0);
assert!(hint.upper().is_none());
let mut hint = SizeHint::new();
hint.set_lower(100);
assert_eq!(hint.lower(), 100);
assert!(hint.upper().is_none());
let mut hint = SizeHint::new();
hint.set_upper(200);
assert_eq!(hint.lower(), 0);
assert_eq!(hint.upper(), Some(200));
let mut hint = SizeHint::new();
hint.set_lower(100);
hint.set_upper(100);
assert_eq!(hint.lower(), 100);
assert_eq!(hint.upper(), Some(100));
}
#[test]
#[should_panic]
fn size_hint_lower_bigger_than_upper() {
let mut hint = SizeHint::new();
hint.set_upper(100);
hint.set_lower(200);
}
#[test]
#[should_panic]
fn size_hint_upper_less_than_lower() {
let mut hint = SizeHint::new();
hint.set_lower(200);
hint.set_upper(100);
}
-49
View File
@@ -1,49 +0,0 @@
extern crate bytes;
extern crate futures;
extern crate tokio_buf;
extern crate tokio_mock_task;
use futures::sync::mpsc;
use futures::Async::*;
use std::io::Cursor;
use tokio_buf::{util, BufStream};
use tokio_mock_task::MockTask;
#[macro_use]
mod support;
type Buf = Cursor<&'static [u8]>;
#[test]
fn empty_stream() {
let (_, rx) = mpsc::unbounded::<Buf>();
let mut bs = util::stream(rx);
assert_none!(bs.poll_buf());
}
#[test]
fn full_stream() {
let (tx, rx) = mpsc::unbounded();
let mut bs = util::stream(rx);
let mut task = MockTask::new();
tx.unbounded_send(buf(b"one")).unwrap();
assert_buf_eq!(bs.poll_buf(), "one");
task.enter(|| assert_not_ready!(bs.poll_buf()));
tx.unbounded_send(buf(b"two")).unwrap();
assert!(task.is_notified());
assert_buf_eq!(bs.poll_buf(), "two");
task.enter(|| assert_not_ready!(bs.poll_buf()));
drop(tx);
assert!(task.is_notified());
assert_none!(bs.poll_buf());
}
fn buf(data: &'static [u8]) -> Buf {
Cursor::new(data)
}
-38
View File
@@ -1,38 +0,0 @@
extern crate bytes;
extern crate futures;
extern crate tokio_buf;
use futures::Async::*;
use std::fmt;
use tokio_buf::BufStream;
#[macro_use]
mod support;
fn test_hello_world<B>(mut bs: B)
where
B: BufStream + fmt::Debug,
B::Item: fmt::Debug,
B::Error: fmt::Debug,
{
let hint = bs.size_hint();
assert_eq!(hint.lower(), 11);
assert_eq!(hint.upper(), Some(11));
assert_buf_eq!(bs.poll_buf(), "hello world");
let hint = bs.size_hint();
assert_eq!(hint.lower(), 0);
assert_eq!(hint.upper(), Some(0));
assert_none!(bs.poll_buf());
}
#[test]
fn string() {
test_hello_world("hello world".to_string());
}
#[test]
fn str() {
test_hello_world("hello world");
}
-133
View File
@@ -1,133 +0,0 @@
#![allow(unused)]
extern crate bytes;
extern crate futures;
extern crate tokio_buf;
use bytes::Buf;
use futures::Async::*;
use futures::Poll;
use tokio_buf::{BufStream, SizeHint};
use std::collections::VecDeque;
use std::io::Cursor;
macro_rules! assert_buf_eq {
($actual:expr, $expect:expr) => {{
use bytes::Buf;
match $actual {
Ok(Ready(Some(val))) => {
assert_eq!(val.remaining(), val.bytes().len());
assert_eq!(val.bytes(), $expect.as_bytes());
}
Ok(Ready(None)) => panic!("expected value; BufStream yielded None"),
Ok(NotReady) => panic!("expected value; BufStream is not ready"),
Err(e) => panic!("expected value; got error = {:?}", e),
}
}};
}
macro_rules! assert_none {
($actual:expr) => {
match $actual {
Ok(Ready(None)) => {}
actual => panic!("expected None; actual = {:?}", actual),
}
};
}
macro_rules! assert_not_ready {
($actual:expr) => {
match $actual {
Ok(NotReady) => {}
actual => panic!("expected NotReady; actual = {:?}", actual),
}
};
}
// ===== Test utils =====
pub fn one(buf: &'static str) -> Mock {
list(&[buf])
}
pub fn list(bufs: &[&'static str]) -> Mock {
let mut polls = VecDeque::new();
for &buf in bufs {
polls.push_back(Ok(Ready(buf.as_bytes())));
}
Mock {
polls,
size_hint: SizeHint::default(),
}
}
pub fn new_mock(values: &[Poll<&'static str, ()>]) -> Mock {
let mut polls = VecDeque::new();
for &v in values {
polls.push_back(match v {
Ok(Ready(v)) => Ok(Ready(v.as_bytes())),
Ok(NotReady) => Ok(NotReady),
Err(e) => Err(e),
});
}
Mock {
polls,
size_hint: SizeHint::default(),
}
}
#[derive(Debug)]
pub struct Mock {
pub polls: VecDeque<Poll<&'static [u8], ()>>,
pub size_hint: SizeHint,
}
#[derive(Debug)]
pub struct MockBuf {
pub data: Cursor<&'static [u8]>,
}
impl BufStream for Mock {
type Item = MockBuf;
type Error = ();
fn poll_buf(&mut self) -> Poll<Option<Self::Item>, Self::Error> {
match self.polls.pop_front() {
Some(Ok(Ready(value))) => Ok(Ready(Some(MockBuf::new(value)))),
Some(Ok(NotReady)) => Ok(NotReady),
Some(Err(e)) => Err(e),
None => Ok(Ready(None)),
}
}
fn size_hint(&self) -> SizeHint {
self.size_hint.clone()
}
}
impl MockBuf {
fn new(data: &'static [u8]) -> MockBuf {
MockBuf {
data: Cursor::new(data),
}
}
}
impl Buf for MockBuf {
fn remaining(&self) -> usize {
self.data.remaining()
}
fn bytes(&self) -> &[u8] {
self.data.bytes()
}
fn advance(&mut self, cnt: usize) {
self.data.advance(cnt)
}
}
-7
View File
@@ -1,7 +0,0 @@
# 0.1.1 (September 26, 2018)
* Allow setting max line length with `LinesCodec` (#632)
# 0.1.0 (June 13, 2018)
* Initial release (#353)
-25
View File
@@ -1,25 +0,0 @@
[package]
name = "tokio-codec"
# When releasing to crates.io:
# - Remove path dependencies
# - Update html_root_url.
# - Update doc url
# - Cargo.toml
# - README.md
# - Update CHANGELOG.md.
# - Create "v0.1.x" git tag.
version = "0.1.1"
authors = ["Carl Lerche <[email protected]>", "Bryan Burgers <[email protected]>"]
license = "MIT"
repository = "https://github.com/tokio-rs/tokio"
homepage = "https://tokio.rs"
documentation = "https://docs.rs/tokio-codec/0.1.1/tokio_codec"
description = """
Utilities for encoding and decoding frames.
"""
categories = ["asynchronous"]
[dependencies]
tokio-io = "0.1.7"
bytes = "0.4.7"
futures = "0.1.18"
-25
View File
@@ -1,25 +0,0 @@
Copyright (c) 2019 Tokio Contributors
Permission is hereby granted, free of charge, to any
person obtaining a copy of this software and associated
documentation files (the "Software"), to deal in the
Software without restriction, including without
limitation the rights to use, copy, modify, merge,
publish, distribute, sublicense, and/or sell copies of
the Software, and to permit persons to whom the Software
is furnished to do so, subject to the following
conditions:
The above copyright notice and this permission notice
shall be included in all copies or substantial portions
of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF
ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED
TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A
PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT
SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY
CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR
IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
DEALINGS IN THE SOFTWARE.
-35
View File
@@ -1,35 +0,0 @@
# tokio-codec
Utilities for encoding and decoding frames.
[Documentation](https://docs.rs/tokio-codec)
## Usage
First, add this to your `Cargo.toml`:
```toml
[dependencies]
tokio-codec = "0.1"
```
Next, add this to your crate:
```rust
extern crate tokio_codec;
```
You can find extensive documentation and examples about how to use this crate
online at [https://tokio.rs](https://tokio.rs). The [API
documentation](https://docs.rs/tokio-codec) is also a great place to get started
for the nitty-gritty.
## License
This project is licensed under the [MIT license](LICENSE).
### Contribution
Unless you explicitly state otherwise, any contribution intentionally submitted
for inclusion in Tokio by you, shall be licensed as MIT, without any additional
terms or conditions.
-25
View File
@@ -1,25 +0,0 @@
#![deny(missing_docs, missing_debug_implementations, warnings)]
#![doc(html_root_url = "https://docs.rs/tokio-codec/0.1.1")]
//! Utilities for encoding and decoding frames.
//!
//! Contains adapters to go from streams of bytes, [`AsyncRead`] and
//! [`AsyncWrite`], to framed streams implementing [`Sink`] and [`Stream`].
//! Framed streams are also known as [transports].
//!
//! [`AsyncRead`]: #
//! [`AsyncWrite`]: #
//! [`Sink`]: #
//! [`Stream`]: #
//! [transports]: #
extern crate bytes;
extern crate tokio_io;
mod bytes_codec;
mod lines_codec;
pub use tokio_io::_tokio_codec::{Decoder, Encoder, Framed, FramedParts, FramedRead, FramedWrite};
pub use bytes_codec::BytesCodec;
pub use lines_codec::LinesCodec;
-215
View File
@@ -1,215 +0,0 @@
extern crate bytes;
extern crate futures;
extern crate tokio_codec;
extern crate tokio_io;
use tokio_codec::{Decoder, FramedRead};
use tokio_io::AsyncRead;
use bytes::{Buf, BytesMut, IntoBuf};
use futures::Async::{NotReady, Ready};
use futures::Stream;
use std::collections::VecDeque;
use std::io::{self, Read};
macro_rules! mock {
($($x:expr,)*) => {{
let mut v = VecDeque::new();
v.extend(vec![$($x),*]);
Mock { calls: v }
}};
}
struct U32Decoder;
impl Decoder for U32Decoder {
type Item = u32;
type Error = io::Error;
fn decode(&mut self, buf: &mut BytesMut) -> io::Result<Option<u32>> {
if buf.len() < 4 {
return Ok(None);
}
let n = buf.split_to(4).into_buf().get_u32_be();
Ok(Some(n))
}
}
#[test]
fn read_multi_frame_in_packet() {
let mock = mock! {
Ok(b"\x00\x00\x00\x00\x00\x00\x00\x01\x00\x00\x00\x02".to_vec()),
};
let mut framed = FramedRead::new(mock, U32Decoder);
assert_eq!(Ready(Some(0)), framed.poll().unwrap());
assert_eq!(Ready(Some(1)), framed.poll().unwrap());
assert_eq!(Ready(Some(2)), framed.poll().unwrap());
assert_eq!(Ready(None), framed.poll().unwrap());
}
#[test]
fn read_multi_frame_across_packets() {
let mock = mock! {
Ok(b"\x00\x00\x00\x00".to_vec()),
Ok(b"\x00\x00\x00\x01".to_vec()),
Ok(b"\x00\x00\x00\x02".to_vec()),
};
let mut framed = FramedRead::new(mock, U32Decoder);
assert_eq!(Ready(Some(0)), framed.poll().unwrap());
assert_eq!(Ready(Some(1)), framed.poll().unwrap());
assert_eq!(Ready(Some(2)), framed.poll().unwrap());
assert_eq!(Ready(None), framed.poll().unwrap());
}
#[test]
fn read_not_ready() {
let mock = mock! {
Err(io::Error::new(io::ErrorKind::WouldBlock, "")),
Ok(b"\x00\x00\x00\x00".to_vec()),
Ok(b"\x00\x00\x00\x01".to_vec()),
};
let mut framed = FramedRead::new(mock, U32Decoder);
assert_eq!(NotReady, framed.poll().unwrap());
assert_eq!(Ready(Some(0)), framed.poll().unwrap());
assert_eq!(Ready(Some(1)), framed.poll().unwrap());
assert_eq!(Ready(None), framed.poll().unwrap());
}
#[test]
fn read_partial_then_not_ready() {
let mock = mock! {
Ok(b"\x00\x00".to_vec()),
Err(io::Error::new(io::ErrorKind::WouldBlock, "")),
Ok(b"\x00\x00\x00\x00\x00\x01\x00\x00\x00\x02".to_vec()),
};
let mut framed = FramedRead::new(mock, U32Decoder);
assert_eq!(NotReady, framed.poll().unwrap());
assert_eq!(Ready(Some(0)), framed.poll().unwrap());
assert_eq!(Ready(Some(1)), framed.poll().unwrap());
assert_eq!(Ready(Some(2)), framed.poll().unwrap());
assert_eq!(Ready(None), framed.poll().unwrap());
}
#[test]
fn read_err() {
let mock = mock! {
Err(io::Error::new(io::ErrorKind::Other, "")),
};
let mut framed = FramedRead::new(mock, U32Decoder);
assert_eq!(io::ErrorKind::Other, framed.poll().unwrap_err().kind());
}
#[test]
fn read_partial_then_err() {
let mock = mock! {
Ok(b"\x00\x00".to_vec()),
Err(io::Error::new(io::ErrorKind::Other, "")),
};
let mut framed = FramedRead::new(mock, U32Decoder);
assert_eq!(io::ErrorKind::Other, framed.poll().unwrap_err().kind());
}
#[test]
fn read_partial_would_block_then_err() {
let mock = mock! {
Ok(b"\x00\x00".to_vec()),
Err(io::Error::new(io::ErrorKind::WouldBlock, "")),
Err(io::Error::new(io::ErrorKind::Other, "")),
};
let mut framed = FramedRead::new(mock, U32Decoder);
assert_eq!(NotReady, framed.poll().unwrap());
assert_eq!(io::ErrorKind::Other, framed.poll().unwrap_err().kind());
}
#[test]
fn huge_size() {
let data = [0; 32 * 1024];
let mut framed = FramedRead::new(&data[..], BigDecoder);
assert_eq!(Ready(Some(0)), framed.poll().unwrap());
assert_eq!(Ready(None), framed.poll().unwrap());
struct BigDecoder;
impl Decoder for BigDecoder {
type Item = u32;
type Error = io::Error;
fn decode(&mut self, buf: &mut BytesMut) -> io::Result<Option<u32>> {
if buf.len() < 32 * 1024 {
return Ok(None);
}
buf.split_to(32 * 1024);
Ok(Some(0))
}
}
}
#[test]
fn data_remaining_is_error() {
let data = [0; 5];
let mut framed = FramedRead::new(&data[..], U32Decoder);
assert_eq!(Ready(Some(0)), framed.poll().unwrap());
assert!(framed.poll().is_err());
}
#[test]
fn multi_frames_on_eof() {
struct MyDecoder(Vec<u32>);
impl Decoder for MyDecoder {
type Item = u32;
type Error = io::Error;
fn decode(&mut self, _buf: &mut BytesMut) -> io::Result<Option<u32>> {
unreachable!();
}
fn decode_eof(&mut self, _buf: &mut BytesMut) -> io::Result<Option<u32>> {
if self.0.is_empty() {
return Ok(None);
}
Ok(Some(self.0.remove(0)))
}
}
let mut framed = FramedRead::new(mock!(), MyDecoder(vec![0, 1, 2, 3]));
assert_eq!(Ready(Some(0)), framed.poll().unwrap());
assert_eq!(Ready(Some(1)), framed.poll().unwrap());
assert_eq!(Ready(Some(2)), framed.poll().unwrap());
assert_eq!(Ready(Some(3)), framed.poll().unwrap());
assert_eq!(Ready(None), framed.poll().unwrap());
}
// ===== Mock ======
struct Mock {
calls: VecDeque<io::Result<Vec<u8>>>,
}
impl Read for Mock {
fn read(&mut self, dst: &mut [u8]) -> io::Result<usize> {
match self.calls.pop_front() {
Some(Ok(data)) => {
debug_assert!(dst.len() >= data.len());
dst[..data.len()].copy_from_slice(&data[..]);
Ok(data.len())
}
Some(Err(e)) => Err(e),
None => Ok(0),
}
}
}
impl AsyncRead for Mock {}
-134
View File
@@ -1,134 +0,0 @@
extern crate bytes;
extern crate futures;
extern crate tokio_codec;
extern crate tokio_io;
use tokio_codec::{Encoder, FramedWrite};
use tokio_io::AsyncWrite;
use bytes::{BufMut, BytesMut};
use futures::{Poll, Sink};
use std::collections::VecDeque;
use std::io::{self, Write};
macro_rules! mock {
($($x:expr,)*) => {{
let mut v = VecDeque::new();
v.extend(vec![$($x),*]);
Mock { calls: v }
}};
}
struct U32Encoder;
impl Encoder for U32Encoder {
type Item = u32;
type Error = io::Error;
fn encode(&mut self, item: u32, dst: &mut BytesMut) -> io::Result<()> {
// Reserve space
dst.reserve(4);
dst.put_u32_be(item);
Ok(())
}
}
#[test]
fn write_multi_frame_in_packet() {
let mock = mock! {
Ok(b"\x00\x00\x00\x00\x00\x00\x00\x01\x00\x00\x00\x02".to_vec()),
};
let mut framed = FramedWrite::new(mock, U32Encoder);
assert!(framed.start_send(0).unwrap().is_ready());
assert!(framed.start_send(1).unwrap().is_ready());
assert!(framed.start_send(2).unwrap().is_ready());
// Nothing written yet
assert_eq!(1, framed.get_ref().calls.len());
// Flush the writes
assert!(framed.poll_complete().unwrap().is_ready());
assert_eq!(0, framed.get_ref().calls.len());
}
#[test]
fn write_hits_backpressure() {
const ITER: usize = 2 * 1024;
let mut mock = mock! {
// Block the `ITER`th write
Err(io::Error::new(io::ErrorKind::WouldBlock, "not ready")),
Ok(b"".to_vec()),
};
for i in 0..(ITER + 1) {
let mut b = BytesMut::with_capacity(4);
b.put_u32_be(i as u32);
// Append to the end
match mock.calls.back_mut().unwrap() {
&mut Ok(ref mut data) => {
// Write in 2kb chunks
if data.len() < ITER {
data.extend_from_slice(&b[..]);
continue;
}
}
_ => unreachable!(),
}
// Push a new new chunk
mock.calls.push_back(Ok(b[..].to_vec()));
}
let mut framed = FramedWrite::new(mock, U32Encoder);
for i in 0..ITER {
assert!(framed.start_send(i as u32).unwrap().is_ready());
}
// This should reject
assert!(!framed.start_send(ITER as u32).unwrap().is_ready());
// This should succeed and start flushing the buffer.
assert!(framed.start_send(ITER as u32).unwrap().is_ready());
// Flush the rest of the buffer
assert!(framed.poll_complete().unwrap().is_ready());
// Ensure the mock is empty
assert_eq!(0, framed.get_ref().calls.len());
}
// ===== Mock ======
struct Mock {
calls: VecDeque<io::Result<Vec<u8>>>,
}
impl Write for Mock {
fn write(&mut self, src: &[u8]) -> io::Result<usize> {
match self.calls.pop_front() {
Some(Ok(data)) => {
assert!(src.len() >= data.len());
assert_eq!(&data[..], &src[..data.len()]);
Ok(data.len())
}
Some(Err(e)) => Err(e),
None => panic!("unexpected write; {:?}", src),
}
}
fn flush(&mut self) -> io::Result<()> {
Ok(())
}
}
impl AsyncWrite for Mock {
fn shutdown(&mut self) -> Poll<(), io::Error> {
Ok(().into())
}
}
-31
View File
@@ -1,31 +0,0 @@
# 0.1.6 (March 22, 2019)
### Added
- implement `TypedExecutor` (#993).
# 0.1.5 (March 1, 2019)
### Fixed
- Documentation typos (#882).
# 0.1.4 (November 21, 2018)
* Fix shutdown on idle (#763).
# 0.1.3 (September 27, 2018)
* Fix minimal versions
# 0.1.2 (September 26, 2018)
* Implement `futures::Executor` for executor types (#563)
* Spawning performance improvements (#565)
# 0.1.1 (August 6, 2018)
* Implement `std::Error` for misc error types (#501)
* bugfix: Track tasks pending in spawn queue (#478)
# 0.1.0 (June 13, 2018)
* Extract `tokio::executor::current_thread` to a tokio-current-thread crate (#356)
-25
View File
@@ -1,25 +0,0 @@
[package]
name = "tokio-current-thread"
# When releasing to crates.io:
# - Remove path dependencies
# - Update html_root_url.
# - Update doc url
# - Cargo.toml
# - README.md
# - Update CHANGELOG.md.
# - Create "v0.1.x" git tag.
version = "0.1.6"
documentation = "https://docs.rs/tokio-current-thread/0.1.6/tokio_current_thread"
repository = "https://github.com/tokio-rs/tokio"
homepage = "https://github.com/tokio-rs/tokio"
license = "MIT"
authors = ["Carl Lerche <[email protected]>"]
description = """
Single threaded executor which manage many tasks concurrently on the current thread.
"""
keywords = ["futures", "tokio"]
categories = ["concurrency", "asynchronous"]
[dependencies]
tokio-executor = "0.1.7"
futures = "0.1.19"
-25
View File
@@ -1,25 +0,0 @@
Copyright (c) 2019 Tokio Contributors
Permission is hereby granted, free of charge, to any
person obtaining a copy of this software and associated
documentation files (the "Software"), to deal in the
Software without restriction, including without
limitation the rights to use, copy, modify, merge,
publish, distribute, sublicense, and/or sell copies of
the Software, and to permit persons to whom the Software
is furnished to do so, subject to the following
conditions:
The above copyright notice and this permission notice
shall be included in all copies or substantial portions
of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF
ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED
TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A
PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT
SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY
CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR
IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
DEALINGS IN THE SOFTWARE.
-19
View File
@@ -1,19 +0,0 @@
# tokio-current-thread
Single threaded executor for Tokio.
[Documentation](https://docs.rs/tokio-current-thread/0.1.6/tokio_current_thread/)
## Overview
This crate provides the single threaded executor which execute many tasks concurrently.
## License
This project is licensed under the [MIT license](LICENSE).
### Contribution
Unless you explicitly state otherwise, any contribution intentionally submitted
for inclusion in Tokio by you, shall be licensed as MIT, without any additional
terms or conditions.
-866
View File
@@ -1,866 +0,0 @@
#![doc(html_root_url = "https://docs.rs/tokio-current-thread/0.1.6")]
#![deny(warnings, missing_docs, missing_debug_implementations)]
//! A single-threaded executor which executes tasks on the same thread from which
//! they are spawned.
//!
//!
//! The crate provides:
//!
//! * [`CurrentThread`] is the main type of this crate. It executes tasks on the current thread.
//! The easiest way to start a new [`CurrentThread`] executor is to call
//! [`block_on_all`] with an initial task to seed the executor.
//! All tasks that are being managed by a [`CurrentThread`] executor are able to
//! spawn additional tasks by calling [`spawn`].
//!
//!
//! Application authors will not use this crate directly. Instead, they will use the
//! `tokio` crate. Library authors should only depend on `tokio-current-thread` if they
//! are building a custom task executor.
//!
//! For more details, see [executor module] documentation in the Tokio crate.
//!
//! [`CurrentThread`]: struct.CurrentThread.html
//! [`spawn`]: fn.spawn.html
//! [`block_on_all`]: fn.block_on_all.html
//! [executor module]: https://docs.rs/tokio/0.1/tokio/executor/index.html
extern crate futures;
extern crate tokio_executor;
mod scheduler;
use self::scheduler::Scheduler;
use tokio_executor::park::{Park, ParkThread, Unpark};
use tokio_executor::{Enter, SpawnError};
use futures::future::{ExecuteError, ExecuteErrorKind, Executor};
use futures::{executor, Async, Future};
use std::cell::Cell;
use std::error::Error;
use std::fmt;
use std::rc::Rc;
use std::sync::{atomic, mpsc, Arc};
use std::thread;
use std::time::{Duration, Instant};
/// Executes tasks on the current thread
pub struct CurrentThread<P: Park = ParkThread> {
/// Execute futures and receive unpark notifications.
scheduler: Scheduler<P::Unpark>,
/// Current number of futures being executed.
///
/// The LSB is used to indicate that the runtime is preparing to shut down.
/// Thus, to get the actual number of pending futures, `>>1`.
num_futures: Arc<atomic::AtomicUsize>,
/// Thread park handle
park: P,
/// Handle for spawning new futures from other threads
spawn_handle: Handle,
/// Receiver for futures spawned from other threads
spawn_receiver: mpsc::Receiver<Box<Future<Item = (), Error = ()> + Send + 'static>>,
/// The thread-local ID assigned to this executor.
id: u64,
}
/// Executes futures on the current thread.
///
/// All futures executed using this executor will be executed on the current
/// thread. As such, `run` will wait for these futures to complete before
/// returning.
///
/// For more details, see the [module level](index.html) documentation.
#[derive(Debug, Clone)]
pub struct TaskExecutor {
// Prevent the handle from moving across threads.
_p: ::std::marker::PhantomData<Rc<()>>,
}
/// Returned by the `turn` function.
#[derive(Debug)]
pub struct Turn {
polled: bool,
}
impl Turn {
/// `true` if any futures were polled at all and `false` otherwise.
pub fn has_polled(&self) -> bool {
self.polled
}
}
/// A `CurrentThread` instance bound to a supplied execution context.
pub struct Entered<'a, P: Park + 'a> {
executor: &'a mut CurrentThread<P>,
enter: &'a mut Enter,
}
/// Error returned by the `run` function.
#[derive(Debug)]
pub struct RunError {
_p: (),
}
impl fmt::Display for RunError {
fn fmt(&self, fmt: &mut fmt::Formatter) -> fmt::Result {
write!(fmt, "{}", self.description())
}
}
impl Error for RunError {
fn description(&self) -> &str {
"Run error"
}
}
/// Error returned by the `run_timeout` function.
#[derive(Debug)]
pub struct RunTimeoutError {
timeout: bool,
}
impl fmt::Display for RunTimeoutError {
fn fmt(&self, fmt: &mut fmt::Formatter) -> fmt::Result {
write!(fmt, "{}", self.description())
}
}
impl Error for RunTimeoutError {
fn description(&self) -> &str {
if self.timeout {
"Run timeout error (timeout)"
} else {
"Run timeout error (not timeout)"
}
}
}
/// Error returned by the `turn` function.
#[derive(Debug)]
pub struct TurnError {
_p: (),
}
impl fmt::Display for TurnError {
fn fmt(&self, fmt: &mut fmt::Formatter) -> fmt::Result {
write!(fmt, "{}", self.description())
}
}
impl Error for TurnError {
fn description(&self) -> &str {
"Turn error"
}
}
/// Error returned by the `block_on` function.
#[derive(Debug)]
pub struct BlockError<T> {
inner: Option<T>,
}
impl<T> fmt::Display for BlockError<T> {
fn fmt(&self, fmt: &mut fmt::Formatter) -> fmt::Result {
write!(fmt, "Block error")
}
}
impl<T: fmt::Debug> Error for BlockError<T> {
fn description(&self) -> &str {
"Block error"
}
}
/// This is mostly split out to make the borrow checker happy.
struct Borrow<'a, U: 'a> {
id: u64,
scheduler: &'a mut Scheduler<U>,
num_futures: &'a atomic::AtomicUsize,
}
trait SpawnLocal {
fn spawn_local(&mut self, future: Box<Future<Item = (), Error = ()>>, already_counted: bool);
}
struct CurrentRunner {
spawn: Cell<Option<*mut SpawnLocal>>,
id: Cell<Option<u64>>,
}
thread_local! {
/// Current thread's task runner. This is set in `TaskRunner::with`
static CURRENT: CurrentRunner = CurrentRunner {
spawn: Cell::new(None),
id: Cell::new(None),
}
}
thread_local! {
/// Unique ID to assign to each new executor launched on this thread.
///
/// The unique ID is used to determine if the currently running executor matches the one
/// referred to by a `Handle` so that direct task dispatch can be used.
static EXECUTOR_ID: Cell<u64> = Cell::new(0)
}
/// Run the executor bootstrapping the execution with the provided future.
///
/// This creates a new [`CurrentThread`] executor, spawns the provided future,
/// and blocks the current thread until the provided future and **all**
/// subsequently spawned futures complete. In other words:
///
/// * If the provided bootstrap future does **not** spawn any additional tasks,
/// `block_on_all` returns once `future` completes.
/// * If the provided bootstrap future **does** spawn additional tasks, then
/// `block_on_all` returns once **all** spawned futures complete.
///
/// See [module level][mod] documentation for more details.
///
/// [`CurrentThread`]: struct.CurrentThread.html
/// [mod]: index.html
pub fn block_on_all<F>(future: F) -> Result<F::Item, F::Error>
where
F: Future,
{
let mut current_thread = CurrentThread::new();
let ret = current_thread.block_on(future);
current_thread.run().unwrap();
ret.map_err(|e| e.into_inner().expect("unexpected execution error"))
}
/// Executes a future on the current thread.
///
/// The provided future must complete or be canceled before `run` will return.
///
/// Unlike [`tokio::spawn`], this function will always spawn on a
/// `CurrentThread` executor and is able to spawn futures that are not `Send`.
///
/// # Panics
///
/// This function can only be invoked from the context of a `run` call; any
/// other use will result in a panic.
///
/// [`tokio::spawn`]: ../fn.spawn.html
pub fn spawn<F>(future: F)
where
F: Future<Item = (), Error = ()> + 'static,
{
TaskExecutor::current()
.spawn_local(Box::new(future))
.unwrap();
}
// ===== impl CurrentThread =====
impl CurrentThread<ParkThread> {
/// Create a new instance of `CurrentThread`.
pub fn new() -> Self {
CurrentThread::new_with_park(ParkThread::new())
}
}
impl<P: Park> CurrentThread<P> {
/// Create a new instance of `CurrentThread` backed by the given park
/// handle.
pub fn new_with_park(park: P) -> Self {
let unpark = park.unpark();
let (spawn_sender, spawn_receiver) = mpsc::channel();
let thread = thread::current().id();
let id = EXECUTOR_ID.with(|idc| {
let id = idc.get();
idc.set(id + 1);
id
});
let scheduler = Scheduler::new(unpark);
let notify = scheduler.notify();
let num_futures = Arc::new(atomic::AtomicUsize::new(0));
CurrentThread {
scheduler: scheduler,
num_futures: num_futures.clone(),
park,
id,
spawn_handle: Handle {
sender: spawn_sender,
num_futures: num_futures,
notify: notify,
shut_down: Cell::new(false),
thread: thread,
id,
},
spawn_receiver: spawn_receiver,
}
}
/// Returns `true` if the executor is currently idle.
///
/// An idle executor is defined by not currently having any spawned tasks.
///
/// Note that this method is inherently racy -- if a future is spawned from a remote `Handle`,
/// this method may return `true` even though there are more futures to be executed.
pub fn is_idle(&self) -> bool {
self.num_futures.load(atomic::Ordering::SeqCst) <= 1
}
/// Spawn the future on the executor.
///
/// This internally queues the future to be executed once `run` is called.
pub fn spawn<F>(&mut self, future: F) -> &mut Self
where
F: Future<Item = (), Error = ()> + 'static,
{
self.borrow().spawn_local(Box::new(future), false);
self
}
/// Synchronously waits for the provided `future` to complete.
///
/// This function can be used to synchronously block the current thread
/// until the provided `future` has resolved either successfully or with an
/// error. The result of the future is then returned from this function
/// call.
///
/// Note that this function will **also** execute any spawned futures on the
/// current thread, but will **not** block until these other spawned futures
/// have completed.
///
/// The caller is responsible for ensuring that other spawned futures
/// complete execution.
pub fn block_on<F>(&mut self, future: F) -> Result<F::Item, BlockError<F::Error>>
where
F: Future,
{
let mut enter = tokio_executor::enter().expect("failed to start `current_thread::Runtime`");
self.enter(&mut enter).block_on(future)
}
/// Run the executor to completion, blocking the thread until **all**
/// spawned futures have completed.
pub fn run(&mut self) -> Result<(), RunError> {
let mut enter = tokio_executor::enter().expect("failed to start `current_thread::Runtime`");
self.enter(&mut enter).run()
}
/// Run the executor to completion, blocking the thread until all
/// spawned futures have completed **or** `duration` time has elapsed.
pub fn run_timeout(&mut self, duration: Duration) -> Result<(), RunTimeoutError> {
let mut enter = tokio_executor::enter().expect("failed to start `current_thread::Runtime`");
self.enter(&mut enter).run_timeout(duration)
}
/// Perform a single iteration of the event loop.
///
/// This function blocks the current thread even if the executor is idle.
pub fn turn(&mut self, duration: Option<Duration>) -> Result<Turn, TurnError> {
let mut enter = tokio_executor::enter().expect("failed to start `current_thread::Runtime`");
self.enter(&mut enter).turn(duration)
}
/// Bind `CurrentThread` instance with an execution context.
pub fn enter<'a>(&'a mut self, enter: &'a mut Enter) -> Entered<'a, P> {
Entered {
executor: self,
enter,
}
}
/// Returns a reference to the underlying `Park` instance.
pub fn get_park(&self) -> &P {
&self.park
}
/// Returns a mutable reference to the underlying `Park` instance.
pub fn get_park_mut(&mut self) -> &mut P {
&mut self.park
}
fn borrow(&mut self) -> Borrow<P::Unpark> {
Borrow {
id: self.id,
scheduler: &mut self.scheduler,
num_futures: &*self.num_futures,
}
}
/// Get a new handle to spawn futures on the executor
///
/// Different to the executor itself, the handle can be sent to different
/// threads and can be used to spawn futures on the executor.
pub fn handle(&self) -> Handle {
self.spawn_handle.clone()
}
}
impl<P: Park> Drop for CurrentThread<P> {
fn drop(&mut self) {
// Signal to Handles that no more futures can be spawned by setting LSB.
//
// NOTE: this isn't technically necessary since the send on the mpsc will fail once the
// receiver is dropped, but it's useful to illustrate how clean shutdown will be
// implemented (e.g., by setting the LSB).
let pending = self.num_futures.fetch_add(1, atomic::Ordering::SeqCst);
// TODO: We currently ignore any pending futures at the time we shut down.
//
// The "proper" fix for this is to have an explicit shutdown phase (`shutdown_on_idle`)
// which sets LSB (as above) do make Handle::spawn stop working, and then runs until
// num_futures.load() == 1.
let _ = pending;
}
}
impl tokio_executor::Executor for CurrentThread {
fn spawn(
&mut self,
future: Box<Future<Item = (), Error = ()> + Send>,
) -> Result<(), SpawnError> {
self.borrow().spawn_local(future, false);
Ok(())
}
}
impl<T> tokio_executor::TypedExecutor<T> for CurrentThread
where
T: Future<Item = (), Error = ()> + 'static,
{
fn spawn(&mut self, future: T) -> Result<(), SpawnError> {
self.borrow().spawn_local(Box::new(future), false);
Ok(())
}
}
impl<P: Park> fmt::Debug for CurrentThread<P> {
fn fmt(&self, fmt: &mut fmt::Formatter) -> fmt::Result {
fmt.debug_struct("CurrentThread")
.field("scheduler", &self.scheduler)
.field(
"num_futures",
&self.num_futures.load(atomic::Ordering::SeqCst),
)
.finish()
}
}
// ===== impl Entered =====
impl<'a, P: Park> Entered<'a, P> {
/// Spawn the future on the executor.
///
/// This internally queues the future to be executed once `run` is called.
pub fn spawn<F>(&mut self, future: F) -> &mut Self
where
F: Future<Item = (), Error = ()> + 'static,
{
self.executor.borrow().spawn_local(Box::new(future), false);
self
}
/// Synchronously waits for the provided `future` to complete.
///
/// This function can be used to synchronously block the current thread
/// until the provided `future` has resolved either successfully or with an
/// error. The result of the future is then returned from this function
/// call.
///
/// Note that this function will **also** execute any spawned futures on the
/// current thread, but will **not** block until these other spawned futures
/// have completed.
///
/// The caller is responsible for ensuring that other spawned futures
/// complete execution.
pub fn block_on<F>(&mut self, future: F) -> Result<F::Item, BlockError<F::Error>>
where
F: Future,
{
let mut future = executor::spawn(future);
let notify = self.executor.scheduler.notify();
loop {
let res = self
.executor
.borrow()
.enter(self.enter, || future.poll_future_notify(&notify, 0));
match res {
Ok(Async::Ready(e)) => return Ok(e),
Err(e) => return Err(BlockError { inner: Some(e) }),
Ok(Async::NotReady) => {}
}
self.tick();
if let Err(_) = self.executor.park.park() {
return Err(BlockError { inner: None });
}
}
}
/// Run the executor to completion, blocking the thread until **all**
/// spawned futures have completed.
pub fn run(&mut self) -> Result<(), RunError> {
self.run_timeout2(None).map_err(|_| RunError { _p: () })
}
/// Run the executor to completion, blocking the thread until all
/// spawned futures have completed **or** `duration` time has elapsed.
pub fn run_timeout(&mut self, duration: Duration) -> Result<(), RunTimeoutError> {
self.run_timeout2(Some(duration))
}
/// Perform a single iteration of the event loop.
///
/// This function blocks the current thread even if the executor is idle.
pub fn turn(&mut self, duration: Option<Duration>) -> Result<Turn, TurnError> {
let res = if self.executor.scheduler.has_pending_futures() {
self.executor.park.park_timeout(Duration::from_millis(0))
} else {
match duration {
Some(duration) => self.executor.park.park_timeout(duration),
None => self.executor.park.park(),
}
};
if res.is_err() {
return Err(TurnError { _p: () });
}
let polled = self.tick();
Ok(Turn { polled })
}
/// Returns a reference to the underlying `Park` instance.
pub fn get_park(&self) -> &P {
&self.executor.park
}
/// Returns a mutable reference to the underlying `Park` instance.
pub fn get_park_mut(&mut self) -> &mut P {
&mut self.executor.park
}
fn run_timeout2(&mut self, dur: Option<Duration>) -> Result<(), RunTimeoutError> {
if self.executor.is_idle() {
// Nothing to do
return Ok(());
}
let mut time = dur.map(|dur| (Instant::now() + dur, dur));
loop {
self.tick();
if self.executor.is_idle() {
return Ok(());
}
match time {
Some((until, rem)) => {
if let Err(_) = self.executor.park.park_timeout(rem) {
return Err(RunTimeoutError::new(false));
}
let now = Instant::now();
if now >= until {
return Err(RunTimeoutError::new(true));
}
time = Some((until, until - now));
}
None => {
if let Err(_) = self.executor.park.park() {
return Err(RunTimeoutError::new(false));
}
}
}
}
}
/// Returns `true` if any futures were processed
fn tick(&mut self) -> bool {
// Spawn any futures that were spawned from other threads by manually
// looping over the receiver stream
// FIXME: Slightly ugly but needed to make the borrow checker happy
let (mut borrow, spawn_receiver) = (
Borrow {
id: self.executor.id,
scheduler: &mut self.executor.scheduler,
num_futures: &*self.executor.num_futures,
},
&mut self.executor.spawn_receiver,
);
while let Ok(future) = spawn_receiver.try_recv() {
borrow.spawn_local(future, true);
}
// After any pending futures were scheduled, do the actual tick
borrow
.scheduler
.tick(borrow.id, &mut *self.enter, borrow.num_futures)
}
}
impl<'a, P: Park> fmt::Debug for Entered<'a, P> {
fn fmt(&self, fmt: &mut fmt::Formatter) -> fmt::Result {
fmt.debug_struct("Entered")
.field("executor", &self.executor)
.field("enter", &self.enter)
.finish()
}
}
// ===== impl Handle =====
/// Handle to spawn a future on the corresponding `CurrentThread` instance
#[derive(Clone)]
pub struct Handle {
sender: mpsc::Sender<Box<Future<Item = (), Error = ()> + Send + 'static>>,
num_futures: Arc<atomic::AtomicUsize>,
shut_down: Cell<bool>,
notify: executor::NotifyHandle,
thread: thread::ThreadId,
/// The thread-local ID assigned to this Handle's executor.
id: u64,
}
// Manual implementation because the Sender does not implement Debug
impl fmt::Debug for Handle {
fn fmt(&self, fmt: &mut fmt::Formatter) -> fmt::Result {
fmt.debug_struct("Handle")
.field("shut_down", &self.shut_down.get())
.finish()
}
}
impl Handle {
/// Spawn a future onto the `CurrentThread` instance corresponding to this handle
///
/// # Panics
///
/// This function panics if the spawn fails. Failure occurs if the `CurrentThread`
/// instance of the `Handle` does not exist anymore.
pub fn spawn<F>(&self, future: F) -> Result<(), SpawnError>
where
F: Future<Item = (), Error = ()> + Send + 'static,
{
if thread::current().id() == self.thread {
let mut e = TaskExecutor::current();
if e.id() == Some(self.id) {
return e.spawn_local(Box::new(future));
}
}
if self.shut_down.get() {
return Err(SpawnError::shutdown());
}
// NOTE: += 2 since LSB is the shutdown bit
let pending = self.num_futures.fetch_add(2, atomic::Ordering::SeqCst);
if pending % 2 == 1 {
// Bring the count back so we still know when the Runtime is idle.
self.num_futures.fetch_sub(2, atomic::Ordering::SeqCst);
// Once the Runtime is shutting down, we know it won't come back.
self.shut_down.set(true);
return Err(SpawnError::shutdown());
}
self.sender
.send(Box::new(future))
.expect("CurrentThread does not exist anymore");
// use 0 for the id, CurrentThread does not make use of it
self.notify.notify(0);
Ok(())
}
/// Provides a best effort **hint** to whether or not `spawn` will succeed.
///
/// This function may return both false positives **and** false negatives.
/// If `status` returns `Ok`, then a call to `spawn` will *probably*
/// succeed, but may fail. If `status` returns `Err`, a call to `spawn` will
/// *probably* fail, but may succeed.
///
/// This allows a caller to avoid creating the task if the call to `spawn`
/// has a high likelihood of failing.
pub fn status(&self) -> Result<(), SpawnError> {
if self.shut_down.get() {
return Err(SpawnError::shutdown());
}
Ok(())
}
}
// ===== impl TaskExecutor =====
impl TaskExecutor {
/// Returns an executor that executes futures on the current thread.
///
/// The user of `TaskExecutor` must ensure that when a future is submitted,
/// that it is done within the context of a call to `run`.
///
/// For more details, see the [module level](index.html) documentation.
pub fn current() -> TaskExecutor {
TaskExecutor {
_p: ::std::marker::PhantomData,
}
}
/// Get the current executor's thread-local ID.
fn id(&self) -> Option<u64> {
CURRENT.with(|current| current.id.get())
}
/// Spawn a future onto the current `CurrentThread` instance.
pub fn spawn_local(
&mut self,
future: Box<Future<Item = (), Error = ()>>,
) -> Result<(), SpawnError> {
CURRENT.with(|current| match current.spawn.get() {
Some(spawn) => {
unsafe { (*spawn).spawn_local(future, false) };
Ok(())
}
None => Err(SpawnError::shutdown()),
})
}
}
impl tokio_executor::Executor for TaskExecutor {
fn spawn(
&mut self,
future: Box<Future<Item = (), Error = ()> + Send>,
) -> Result<(), SpawnError> {
self.spawn_local(future)
}
}
impl<F> tokio_executor::TypedExecutor<F> for TaskExecutor
where
F: Future<Item = (), Error = ()> + 'static,
{
fn spawn(&mut self, future: F) -> Result<(), SpawnError> {
self.spawn_local(Box::new(future))
}
}
impl<F> Executor<F> for TaskExecutor
where
F: Future<Item = (), Error = ()> + 'static,
{
fn execute(&self, future: F) -> Result<(), ExecuteError<F>> {
CURRENT.with(|current| match current.spawn.get() {
Some(spawn) => {
unsafe { (*spawn).spawn_local(Box::new(future), false) };
Ok(())
}
None => Err(ExecuteError::new(ExecuteErrorKind::Shutdown, future)),
})
}
}
// ===== impl Borrow =====
impl<'a, U: Unpark> Borrow<'a, U> {
fn enter<F, R>(&mut self, _: &mut Enter, f: F) -> R
where
F: FnOnce() -> R,
{
CURRENT.with(|current| {
current.id.set(Some(self.id));
current.set_spawn(self, || f())
})
}
}
impl<'a, U: Unpark> SpawnLocal for Borrow<'a, U> {
fn spawn_local(&mut self, future: Box<Future<Item = (), Error = ()>>, already_counted: bool) {
if !already_counted {
// NOTE: we have a borrow of the Runtime, so we know that it isn't shut down.
// NOTE: += 2 since LSB is the shutdown bit
self.num_futures.fetch_add(2, atomic::Ordering::SeqCst);
}
self.scheduler.schedule(future);
}
}
// ===== impl CurrentRunner =====
impl CurrentRunner {
fn set_spawn<F, R>(&self, spawn: &mut SpawnLocal, f: F) -> R
where
F: FnOnce() -> R,
{
struct Reset<'a>(&'a CurrentRunner);
impl<'a> Drop for Reset<'a> {
fn drop(&mut self) {
self.0.spawn.set(None);
self.0.id.set(None);
}
}
let _reset = Reset(self);
let spawn = unsafe { hide_lt(spawn as *mut SpawnLocal) };
self.spawn.set(Some(spawn));
f()
}
}
unsafe fn hide_lt<'a>(p: *mut (SpawnLocal + 'a)) -> *mut (SpawnLocal + 'static) {
use std::mem;
mem::transmute(p)
}
// ===== impl RunTimeoutError =====
impl RunTimeoutError {
fn new(timeout: bool) -> Self {
RunTimeoutError { timeout }
}
/// Returns `true` if the error was caused by the operation timing out.
pub fn is_timeout(&self) -> bool {
self.timeout
}
}
impl From<tokio_executor::EnterError> for RunTimeoutError {
fn from(_: tokio_executor::EnterError) -> Self {
RunTimeoutError::new(false)
}
}
// ===== impl BlockError =====
impl<T> BlockError<T> {
/// Returns the error yielded by the future being blocked on
pub fn into_inner(self) -> Option<T> {
self.inner
}
}
impl<T> From<tokio_executor::EnterError> for BlockError<T> {
fn from(_: tokio_executor::EnterError) -> Self {
BlockError { inner: None }
}
}
-770
View File
@@ -1,770 +0,0 @@
use super::Borrow;
use tokio_executor::park::Unpark;
use tokio_executor::Enter;
use futures::executor::{self, NotifyHandle, Spawn, UnsafeNotify};
use futures::{Async, Future};
use std::cell::UnsafeCell;
use std::fmt::{self, Debug};
use std::marker::PhantomData;
use std::mem;
use std::ptr;
use std::sync::atomic::Ordering::{AcqRel, Acquire, Relaxed, Release, SeqCst};
use std::sync::atomic::{AtomicBool, AtomicPtr, AtomicUsize};
use std::sync::{Arc, Weak};
use std::thread;
use std::usize;
/// A generic task-aware scheduler.
///
/// This is used both by `FuturesUnordered` and the current-thread executor.
pub struct Scheduler<U> {
inner: Arc<Inner<U>>,
nodes: List<U>,
}
pub struct Notify<'a, U: 'a>(&'a Arc<Node<U>>);
// A linked-list of nodes
struct List<U> {
len: usize,
head: *const Node<U>,
tail: *const Node<U>,
}
// Scheduler is implemented using two linked lists. The first linked list tracks
// all items managed by a `Scheduler`. This list is stored on the `Scheduler`
// struct and is **not** thread safe. The second linked list is an
// implementation of the intrusive MPSC queue algorithm described by
// 1024cores.net and is stored on `Inner`. This linked list can push items to
// the back concurrently but only one consumer may pop from the front. To
// enforce this requirement, all popping will be performed via fns on
// `Scheduler` that take `&mut self`.
//
// When a item is submitted to the set a node is allocated and inserted in
// both linked lists. This means that all insertion operations **must** be
// originated from `Scheduler` with `&mut self` The next call to `tick` will
// (eventually) see this node and call `poll` on the item.
//
// Nodes are wrapped in `Arc` cells which manage the lifetime of the node.
// However, `Arc` handles are sometimes cast to `*const Node` pointers.
// Specifically, when a node is stored in at least one of the two lists
// described above, this represents a logical `Arc` handle. This is how
// `Scheduler` maintains its reference to all nodes it manages. Each
// `NotifyHandle` instance is an `Arc<Node>` as well.
//
// When `Scheduler` drops, it clears the linked list of all nodes that it
// manages. When doing so, it must attempt to decrement the reference count (by
// dropping an Arc handle). However, it can **only** decrement the reference
// count if the node is not currently stored in the mpsc channel. If the node
// **is** "queued" in the mpsc channel, then the arc reference count cannot be
// decremented. Once the node is popped from the mpsc channel, then the final
// arc reference count can be decremented, thus freeing the node.
struct Inner<U> {
// Thread unpark handle
unpark: U,
// Tick number
tick_num: AtomicUsize,
// Head/tail of the readiness queue
head_readiness: AtomicPtr<Node<U>>,
tail_readiness: UnsafeCell<*const Node<U>>,
// Used as part of the mpsc queue algorithm
stub: Arc<Node<U>>,
}
unsafe impl<U: Sync + Send> Send for Inner<U> {}
unsafe impl<U: Sync + Send> Sync for Inner<U> {}
impl<U: Unpark> executor::Notify for Inner<U> {
fn notify(&self, _: usize) {
self.unpark.unpark();
}
}
struct Node<U> {
// The item
item: UnsafeCell<Option<Task>>,
// The tick at which this node was notified
notified_at: AtomicUsize,
// Next pointer for linked list tracking all active nodes
next_all: UnsafeCell<*const Node<U>>,
// Previous node in linked list tracking all active nodes
prev_all: UnsafeCell<*const Node<U>>,
// Next pointer in readiness queue
next_readiness: AtomicPtr<Node<U>>,
// Whether or not this node is currently in the mpsc queue.
queued: AtomicBool,
// Queue that we'll be enqueued to when notified
queue: Weak<Inner<U>>,
}
/// Returned by `Inner::dequeue`, representing either a dequeue success (with
/// the dequeued node), an empty list, or an inconsistent state.
///
/// The inconsistent state is described in more detail at [1024cores], but
/// roughly indicates that a node will be ready to dequeue sometime shortly in
/// the future and the caller should try again soon.
///
/// [1024cores]: http://www.1024cores.net/home/lock-free-algorithms/queues/intrusive-mpsc-node-based-queue
enum Dequeue<U> {
Data(*const Node<U>),
Empty,
Yield,
Inconsistent,
}
/// Wraps a spawned boxed future
struct Task(Spawn<Box<Future<Item = (), Error = ()>>>);
/// A task that is scheduled. `turn` must be called
pub struct Scheduled<'a, U: 'a> {
task: &'a mut Task,
notify: &'a Notify<'a, U>,
done: &'a mut bool,
}
impl<U> Scheduler<U>
where
U: Unpark,
{
/// Constructs a new, empty `Scheduler`
///
/// The returned `Scheduler` does not contain any items and, in this
/// state, `Scheduler::poll` will return `Ok(Async::Ready(None))`.
pub fn new(unpark: U) -> Self {
let stub = Arc::new(Node {
item: UnsafeCell::new(None),
notified_at: AtomicUsize::new(0),
next_all: UnsafeCell::new(ptr::null()),
prev_all: UnsafeCell::new(ptr::null()),
next_readiness: AtomicPtr::new(ptr::null_mut()),
queued: AtomicBool::new(true),
queue: Weak::new(),
});
let stub_ptr = &*stub as *const Node<U>;
let inner = Arc::new(Inner {
unpark,
tick_num: AtomicUsize::new(0),
head_readiness: AtomicPtr::new(stub_ptr as *mut _),
tail_readiness: UnsafeCell::new(stub_ptr),
stub: stub,
});
Scheduler {
inner: inner,
nodes: List::new(),
}
}
pub fn notify(&self) -> NotifyHandle {
self.inner.clone().into()
}
pub fn schedule(&mut self, item: Box<Future<Item = (), Error = ()>>) {
// Get the current scheduler tick
let tick_num = self.inner.tick_num.load(SeqCst);
let node = Arc::new(Node {
item: UnsafeCell::new(Some(Task::new(item))),
notified_at: AtomicUsize::new(tick_num),
next_all: UnsafeCell::new(ptr::null_mut()),
prev_all: UnsafeCell::new(ptr::null_mut()),
next_readiness: AtomicPtr::new(ptr::null_mut()),
queued: AtomicBool::new(true),
queue: Arc::downgrade(&self.inner),
});
// Right now our node has a strong reference count of 1. We transfer
// ownership of this reference count to our internal linked list
// and we'll reclaim ownership through the `unlink` function below.
let ptr = self.nodes.push_back(node);
// We'll need to get the item "into the system" to start tracking it,
// e.g. getting its unpark notifications going to us tracking which
// items are ready. To do that we unconditionally enqueue it for
// polling here.
self.inner.enqueue(ptr);
}
/// Returns `true` if there are currently any pending futures
pub fn has_pending_futures(&mut self) -> bool {
// See function definition for why the unsafe is needed and
// correctly used here
unsafe { self.inner.has_pending_futures() }
}
/// Advance the scheduler state, returning `true` if any futures were
/// processed.
///
/// This function should be called whenever the caller is notified via a
/// wakeup.
pub fn tick(&mut self, eid: u64, enter: &mut Enter, num_futures: &AtomicUsize) -> bool {
let mut ret = false;
let tick = self.inner.tick_num.fetch_add(1, SeqCst).wrapping_add(1);
loop {
let node = match unsafe { self.inner.dequeue(Some(tick)) } {
Dequeue::Empty => {
return ret;
}
Dequeue::Yield => {
self.inner.unpark.unpark();
return ret;
}
Dequeue::Inconsistent => {
thread::yield_now();
continue;
}
Dequeue::Data(node) => node,
};
ret = true;
debug_assert!(node != self.inner.stub());
unsafe {
if (*(*node).item.get()).is_none() {
// The node has already been released. However, while it was
// being released, another thread notified it, which
// resulted in it getting pushed into the mpsc channel.
//
// In this case, we just decrement the ref count.
let node = ptr2arc(node);
assert!((*node.next_all.get()).is_null());
assert!((*node.prev_all.get()).is_null());
continue;
};
// We're going to need to be very careful if the `poll`
// function below panics. We need to (a) not leak memory and
// (b) ensure that we still don't have any use-after-frees. To
// manage this we do a few things:
//
// * This "bomb" here will call `release_node` if dropped
// abnormally. That way we'll be sure the memory management
// of the `node` is managed correctly.
//
// * We unlink the node from our internal queue to preemptively
// assume is is complete (will return Ready or panic), in
// which case we'll want to discard it regardless.
//
struct Bomb<'a, U: Unpark + 'a> {
borrow: &'a mut Borrow<'a, U>,
enter: &'a mut Enter,
node: Option<Arc<Node<U>>>,
}
impl<'a, U: Unpark> Drop for Bomb<'a, U> {
fn drop(&mut self) {
if let Some(node) = self.node.take() {
self.borrow.enter(self.enter, || release_node(node))
}
}
}
let node = self.nodes.remove(node);
let mut borrow = Borrow {
id: eid,
scheduler: self,
num_futures,
};
let mut bomb = Bomb {
node: Some(node),
enter: enter,
borrow: &mut borrow,
};
let mut done = false;
// Now that the bomb holds the node, create a new scope. This
// scope ensures that the borrow will go out of scope before we
// mutate the node pointer in `bomb` again
{
let node = bomb.node.as_ref().unwrap();
// Get a reference to the inner future. We already ensured
// that the item `is_some`.
let item = (*node.item.get()).as_mut().unwrap();
// Unset queued flag... this must be done before
// polling. This ensures that the item gets
// rescheduled if it is notified **during** a call
// to `poll`.
let prev = (*node).queued.swap(false, SeqCst);
assert!(prev);
// Poll the underlying item with the appropriate `notify`
// implementation. This is where a large bit of the unsafety
// starts to stem from internally. The `notify` instance itself
// is basically just our `Arc<Node>` and tracks the mpsc
// queue of ready items.
//
// Critically though `Node` won't actually access `Task`, the
// item, while it's floating around inside of `Task`
// instances. These structs will basically just use `T` to size
// the internal allocation, appropriately accessing fields and
// deallocating the node if need be.
let borrow = &mut *bomb.borrow;
let enter = &mut *bomb.enter;
let notify = Notify(bomb.node.as_ref().unwrap());
let mut scheduled = Scheduled {
task: item,
notify: &notify,
done: &mut done,
};
if borrow.enter(enter, || scheduled.tick()) {
// we have a borrow of the Runtime, so we know it's not shut down
borrow.num_futures.fetch_sub(2, SeqCst);
}
}
if !done {
// The future is not done, push it back into the "all
// node" list.
let node = bomb.node.take().unwrap();
bomb.borrow.scheduler.nodes.push_back(node);
}
}
}
}
}
impl<'a, U: Unpark> Scheduled<'a, U> {
/// Polls the task, returns `true` if the task has completed.
pub fn tick(&mut self) -> bool {
// Tick the future
let ret = match self.task.0.poll_future_notify(self.notify, 0) {
Ok(Async::Ready(_)) | Err(_) => true,
Ok(Async::NotReady) => false,
};
*self.done = ret;
ret
}
}
impl Task {
pub fn new(future: Box<Future<Item = (), Error = ()> + 'static>) -> Self {
Task(executor::spawn(future))
}
}
impl fmt::Debug for Task {
fn fmt(&self, fmt: &mut fmt::Formatter) -> fmt::Result {
fmt.debug_struct("Task").finish()
}
}
fn release_node<U>(node: Arc<Node<U>>) {
// The item is done, try to reset the queued flag. This will prevent
// `notify` from doing any work in the item
let prev = node.queued.swap(true, SeqCst);
// Drop the item, even if it hasn't finished yet. This is safe
// because we're dropping the item on the thread that owns
// `Scheduler`, which correctly tracks T's lifetimes and such.
unsafe {
drop((*node.item.get()).take());
}
// If the queued flag was previously set then it means that this node
// is still in our internal mpsc queue. We then transfer ownership
// of our reference count to the mpsc queue, and it'll come along and
// free it later, noticing that the item is `None`.
//
// If, however, the queued flag was *not* set then we're safe to
// release our reference count on the internal node. The queued flag
// was set above so all item `enqueue` operations will not actually
// enqueue the node, so our node will never see the mpsc queue again.
// The node itself will be deallocated once all reference counts have
// been dropped by the various owning tasks elsewhere.
if prev {
mem::forget(node);
}
}
impl<U> Debug for Scheduler<U> {
fn fmt(&self, fmt: &mut fmt::Formatter) -> fmt::Result {
write!(fmt, "Scheduler {{ ... }}")
}
}
impl<U> Drop for Scheduler<U> {
fn drop(&mut self) {
// When a `Scheduler` is dropped we want to drop all items associated
// with it. At the same time though there may be tons of `Task` handles
// flying around which contain `Node` references inside them. We'll
// let those naturally get deallocated when the `Task` itself goes out
// of scope or gets notified.
while let Some(node) = self.nodes.pop_front() {
release_node(node);
}
// Note that at this point we could still have a bunch of nodes in the
// mpsc queue. None of those nodes, however, have items associated
// with them so they're safe to destroy on any thread. At this point
// the `Scheduler` struct, the owner of the one strong reference
// to `Inner` will drop the strong reference. At that point
// whichever thread releases the strong refcount last (be it this
// thread or some other thread as part of an `upgrade`) will clear out
// the mpsc queue and free all remaining nodes.
//
// While that freeing operation isn't guaranteed to happen here, it's
// guaranteed to happen "promptly" as no more "blocking work" will
// happen while there's a strong refcount held.
}
}
impl<U> Inner<U> {
/// The enqueue function from the 1024cores intrusive MPSC queue algorithm.
fn enqueue(&self, node: *const Node<U>) {
unsafe {
debug_assert!((*node).queued.load(Relaxed));
// This action does not require any coordination
(*node).next_readiness.store(ptr::null_mut(), Relaxed);
// Note that these atomic orderings come from 1024cores
let node = node as *mut _;
let prev = self.head_readiness.swap(node, AcqRel);
(*prev).next_readiness.store(node, Release);
}
}
/// Returns `true` if there are currently any pending futures
///
/// See `dequeue` for an explanation why this function is unsafe.
unsafe fn has_pending_futures(&self) -> bool {
let tail = *self.tail_readiness.get();
let next = (*tail).next_readiness.load(Acquire);
if tail == self.stub() {
if next.is_null() {
return false;
}
}
true
}
/// The dequeue function from the 1024cores intrusive MPSC queue algorithm
///
/// Note that this unsafe as it required mutual exclusion (only one thread
/// can call this) to be guaranteed elsewhere.
unsafe fn dequeue(&self, tick: Option<usize>) -> Dequeue<U> {
let mut tail = *self.tail_readiness.get();
let mut next = (*tail).next_readiness.load(Acquire);
if tail == self.stub() {
if next.is_null() {
return Dequeue::Empty;
}
*self.tail_readiness.get() = next;
tail = next;
next = (*next).next_readiness.load(Acquire);
}
if let Some(tick) = tick {
let actual = (*tail).notified_at.load(SeqCst);
// Only dequeue if the node was not scheduled during the current
// tick.
if actual == tick {
// Only doing the check above **should** be enough in
// practice. However, technically there is a potential for
// deadlocking if there are `usize::MAX` ticks while the thread
// scheduling the task is frozen.
//
// If, for some reason, this is not enough, calling `unpark`
// here will resolve the issue.
return Dequeue::Yield;
}
}
if !next.is_null() {
*self.tail_readiness.get() = next;
debug_assert!(tail != self.stub());
return Dequeue::Data(tail);
}
if self.head_readiness.load(Acquire) as *const _ != tail {
return Dequeue::Inconsistent;
}
self.enqueue(self.stub());
next = (*tail).next_readiness.load(Acquire);
if !next.is_null() {
*self.tail_readiness.get() = next;
return Dequeue::Data(tail);
}
Dequeue::Inconsistent
}
fn stub(&self) -> *const Node<U> {
&*self.stub
}
}
impl<U> Drop for Inner<U> {
fn drop(&mut self) {
// Once we're in the destructor for `Inner` we need to clear out the
// mpsc queue of nodes if there's anything left in there.
//
// Note that each node has a strong reference count associated with it
// which is owned by the mpsc queue. All nodes should have had their
// items dropped already by the `Scheduler` destructor above,
// so we're just pulling out nodes and dropping their refcounts.
unsafe {
loop {
match self.dequeue(None) {
Dequeue::Empty => break,
Dequeue::Yield => unreachable!(),
Dequeue::Inconsistent => abort("inconsistent in drop"),
Dequeue::Data(ptr) => drop(ptr2arc(ptr)),
}
}
}
}
}
impl<U> List<U> {
fn new() -> Self {
List {
len: 0,
head: ptr::null_mut(),
tail: ptr::null_mut(),
}
}
/// Appends an element to the back of the list
fn push_back(&mut self, node: Arc<Node<U>>) -> *const Node<U> {
let ptr = arc2ptr(node);
unsafe {
// Point to the current last node in the list
*(*ptr).prev_all.get() = self.tail;
*(*ptr).next_all.get() = ptr::null_mut();
if !self.tail.is_null() {
*(*self.tail).next_all.get() = ptr;
self.tail = ptr;
} else {
// This is the first node
self.tail = ptr;
self.head = ptr;
}
}
self.len += 1;
return ptr;
}
/// Pop an element from the front of the list
fn pop_front(&mut self) -> Option<Arc<Node<U>>> {
if self.head.is_null() {
// The list is empty
return None;
}
self.len -= 1;
unsafe {
// Convert the ptr to Arc<_>
let node = ptr2arc(self.head);
// Update the head pointer
self.head = *node.next_all.get();
// If the pointer is null, then the list is empty
if self.head.is_null() {
self.tail = ptr::null_mut();
} else {
*(*self.head).prev_all.get() = ptr::null_mut();
}
Some(node)
}
}
/// Remove a specific node
unsafe fn remove(&mut self, node: *const Node<U>) -> Arc<Node<U>> {
let node = ptr2arc(node);
let next = *node.next_all.get();
let prev = *node.prev_all.get();
*node.next_all.get() = ptr::null_mut();
*node.prev_all.get() = ptr::null_mut();
if !next.is_null() {
*(*next).prev_all.get() = prev;
} else {
self.tail = prev;
}
if !prev.is_null() {
*(*prev).next_all.get() = next;
} else {
self.head = next;
}
self.len -= 1;
return node;
}
}
impl<'a, U> Clone for Notify<'a, U> {
fn clone(&self) -> Self {
Notify(self.0)
}
}
impl<'a, U> fmt::Debug for Notify<'a, U> {
fn fmt(&self, fmt: &mut fmt::Formatter) -> fmt::Result {
fmt.debug_struct("Notify").finish()
}
}
impl<'a, U: Unpark> From<Notify<'a, U>> for NotifyHandle {
fn from(handle: Notify<'a, U>) -> NotifyHandle {
unsafe {
let ptr = handle.0.clone();
let ptr = mem::transmute::<Arc<Node<U>>, *mut ArcNode<U>>(ptr);
NotifyHandle::new(hide_lt(ptr))
}
}
}
struct ArcNode<U>(PhantomData<U>);
// We should never touch `Task` on any thread other than the one owning
// `Scheduler`, so this should be a safe operation.
unsafe impl<U: Sync + Send> Send for ArcNode<U> {}
unsafe impl<U: Sync + Send> Sync for ArcNode<U> {}
impl<U: Unpark> executor::Notify for ArcNode<U> {
fn notify(&self, _id: usize) {
unsafe {
let me: *const ArcNode<U> = self;
let me: *const *const ArcNode<U> = &me;
let me = me as *const Arc<Node<U>>;
Node::notify(&*me)
}
}
}
unsafe impl<U: Unpark> UnsafeNotify for ArcNode<U> {
unsafe fn clone_raw(&self) -> NotifyHandle {
let me: *const ArcNode<U> = self;
let me: *const *const ArcNode<U> = &me;
let me = &*(me as *const Arc<Node<U>>);
Notify(me).into()
}
unsafe fn drop_raw(&self) {
let mut me: *const ArcNode<U> = self;
let me = &mut me as *mut *const ArcNode<U> as *mut Arc<Node<U>>;
ptr::drop_in_place(me);
}
}
unsafe fn hide_lt<U: Unpark>(p: *mut ArcNode<U>) -> *mut UnsafeNotify {
mem::transmute(p as *mut UnsafeNotify)
}
impl<U: Unpark> Node<U> {
fn notify(me: &Arc<Node<U>>) {
let inner = match me.queue.upgrade() {
Some(inner) => inner,
None => return,
};
// It's our job to notify the node that it's ready to get polled,
// meaning that we need to enqueue it into the readiness queue. To
// do this we flag that we're ready to be queued, and if successful
// we then do the literal queueing operation, ensuring that we're
// only queued once.
//
// Once the node is inserted we be sure to notify the parent task,
// as it'll want to come along and pick up our node now.
//
// Note that we don't change the reference count of the node here,
// we're just enqueueing the raw pointer. The `Scheduler`
// implementation guarantees that if we set the `queued` flag true that
// there's a reference count held by the main `Scheduler` queue
// still.
let prev = me.queued.swap(true, SeqCst);
if !prev {
// Get the current scheduler tick
let tick_num = inner.tick_num.load(SeqCst);
me.notified_at.store(tick_num, SeqCst);
inner.enqueue(&**me);
inner.unpark.unpark();
}
}
}
impl<U> Drop for Node<U> {
fn drop(&mut self) {
// Currently a `Node` is sent across all threads for any lifetime,
// regardless of `T`. This means that for memory safety we can't
// actually touch `T` at any time except when we have a reference to the
// `Scheduler` itself.
//
// Consequently it *should* be the case that we always drop items from
// the `Scheduler` instance, but this is a bomb in place to catch
// any bugs in that logic.
unsafe {
if (*self.item.get()).is_some() {
abort("item still here when dropping");
}
}
}
}
fn arc2ptr<T>(ptr: Arc<T>) -> *const T {
let addr = &*ptr as *const T;
mem::forget(ptr);
return addr;
}
unsafe fn ptr2arc<T>(ptr: *const T) -> Arc<T> {
let anchor = mem::transmute::<usize, Arc<T>>(0x10);
let addr = &*anchor as *const T;
mem::forget(anchor);
let offset = addr as isize - 0x10;
mem::transmute::<isize, Arc<T>>(ptr as isize - offset)
}
fn abort(s: &str) -> ! {
struct DoublePanic;
impl Drop for DoublePanic {
fn drop(&mut self) {
panic!("panicking twice to abort the program");
}
}
let _bomb = DoublePanic;
panic!("{}", s);
}
@@ -1,837 +0,0 @@
extern crate futures;
extern crate tokio_current_thread;
extern crate tokio_executor;
use tokio_current_thread::{block_on_all, CurrentThread};
use std::any::Any;
use std::cell::{Cell, RefCell};
use std::rc::Rc;
use std::thread;
use std::time::Duration;
use futures::future::{self, lazy};
use futures::task;
// This is not actually unused --- we need this trait to be in scope for
// the tests that sue TaskExecutor::current().execute(). The compiler
// doesn't realise that.
#[allow(unused_imports)]
use futures::future::Executor as _futures_Executor;
use futures::prelude::*;
use futures::sync::oneshot;
mod from_block_on_all {
use super::*;
fn test<F: Fn(Box<Future<Item = (), Error = ()>>) + 'static>(spawn: F) {
let cnt = Rc::new(Cell::new(0));
let c = cnt.clone();
let msg = tokio_current_thread::block_on_all(lazy(move || {
c.set(1 + c.get());
// Spawn!
spawn(Box::new(lazy(move || {
c.set(1 + c.get());
Ok::<(), ()>(())
})));
Ok::<_, ()>("hello")
}))
.unwrap();
assert_eq!(2, cnt.get());
assert_eq!(msg, "hello");
}
#[test]
fn spawn() {
test(tokio_current_thread::spawn)
}
#[test]
fn execute() {
test(|f| {
tokio_current_thread::TaskExecutor::current()
.execute(f)
.unwrap();
});
}
}
#[test]
fn block_waits() {
let (tx, rx) = oneshot::channel();
thread::spawn(|| {
thread::sleep(Duration::from_millis(1000));
tx.send(()).unwrap();
});
let cnt = Rc::new(Cell::new(0));
let cnt2 = cnt.clone();
block_on_all(rx.then(move |_| {
cnt.set(1 + cnt.get());
Ok::<_, ()>(())
}))
.unwrap();
assert_eq!(1, cnt2.get());
}
#[test]
fn spawn_many() {
const ITER: usize = 200;
let cnt = Rc::new(Cell::new(0));
let mut tokio_current_thread = CurrentThread::new();
for _ in 0..ITER {
let cnt = cnt.clone();
tokio_current_thread.spawn(lazy(move || {
cnt.set(1 + cnt.get());
Ok::<(), ()>(())
}));
}
tokio_current_thread.run().unwrap();
assert_eq!(cnt.get(), ITER);
}
mod does_not_set_global_executor_by_default {
use super::*;
fn test<F: Fn(Box<Future<Item = (), Error = ()> + Send>) -> Result<(), E> + 'static, E>(
spawn: F,
) {
block_on_all(lazy(|| {
spawn(Box::new(lazy(|| ok()))).unwrap_err();
ok()
}))
.unwrap()
}
#[test]
fn spawn() {
use tokio_executor::Executor;
test(|f| tokio_executor::DefaultExecutor::current().spawn(f))
}
#[test]
fn execute() {
test(|f| tokio_executor::DefaultExecutor::current().execute(f))
}
}
mod from_block_on_future {
use super::*;
fn test<F: Fn(Box<Future<Item = (), Error = ()>>)>(spawn: F) {
let cnt = Rc::new(Cell::new(0));
let mut tokio_current_thread = CurrentThread::new();
tokio_current_thread
.block_on(lazy(|| {
let cnt = cnt.clone();
spawn(Box::new(lazy(move || {
cnt.set(1 + cnt.get());
Ok(())
})));
Ok::<_, ()>(())
}))
.unwrap();
tokio_current_thread.run().unwrap();
assert_eq!(1, cnt.get());
}
#[test]
fn spawn() {
test(tokio_current_thread::spawn);
}
#[test]
fn execute() {
test(|f| {
tokio_current_thread::TaskExecutor::current()
.execute(f)
.unwrap();
});
}
}
struct Never(Rc<()>);
impl Future for Never {
type Item = ();
type Error = ();
fn poll(&mut self) -> Poll<(), ()> {
Ok(Async::NotReady)
}
}
mod outstanding_tasks_are_dropped_when_executor_is_dropped {
use super::*;
fn test<F, G>(spawn: F, dotspawn: G)
where
F: Fn(Box<Future<Item = (), Error = ()>>) + 'static,
G: Fn(&mut CurrentThread, Box<Future<Item = (), Error = ()>>),
{
let mut rc = Rc::new(());
let mut tokio_current_thread = CurrentThread::new();
dotspawn(&mut tokio_current_thread, Box::new(Never(rc.clone())));
drop(tokio_current_thread);
// Ensure the daemon is dropped
assert!(Rc::get_mut(&mut rc).is_some());
// Using the global spawn fn
let mut rc = Rc::new(());
let mut tokio_current_thread = CurrentThread::new();
tokio_current_thread
.block_on(lazy(|| {
spawn(Box::new(Never(rc.clone())));
Ok::<_, ()>(())
}))
.unwrap();
drop(tokio_current_thread);
// Ensure the daemon is dropped
assert!(Rc::get_mut(&mut rc).is_some());
}
#[test]
fn spawn() {
test(tokio_current_thread::spawn, |rt, f| {
rt.spawn(f);
})
}
#[test]
fn execute() {
test(
|f| {
tokio_current_thread::TaskExecutor::current()
.execute(f)
.unwrap();
},
// Note: `CurrentThread` doesn't currently implement
// `futures::Executor`, so we'll call `.spawn(...)` rather than
// `.execute(...)` for now. If `CurrentThread` is changed to
// implement Executor, change this to `.execute(...).unwrap()`.
|rt, f| {
rt.spawn(f);
},
);
}
}
#[test]
#[should_panic]
fn nesting_run() {
block_on_all(lazy(|| {
block_on_all(lazy(|| ok())).unwrap();
ok()
}))
.unwrap();
}
mod run_in_future {
use super::*;
#[test]
#[should_panic]
fn spawn() {
block_on_all(lazy(|| {
tokio_current_thread::spawn(lazy(|| {
block_on_all(lazy(|| ok())).unwrap();
ok()
}));
ok()
}))
.unwrap();
}
#[test]
#[should_panic]
fn execute() {
block_on_all(lazy(|| {
tokio_current_thread::TaskExecutor::current()
.execute(lazy(|| {
block_on_all(lazy(|| ok())).unwrap();
ok()
}))
.unwrap();
ok()
}))
.unwrap();
}
}
#[test]
fn tick_on_infini_future() {
let num = Rc::new(Cell::new(0));
struct Infini {
num: Rc<Cell<usize>>,
}
impl Future for Infini {
type Item = ();
type Error = ();
fn poll(&mut self) -> Poll<(), ()> {
self.num.set(1 + self.num.get());
task::current().notify();
Ok(Async::NotReady)
}
}
CurrentThread::new()
.spawn(Infini { num: num.clone() })
.turn(None)
.unwrap();
assert_eq!(1, num.get());
}
mod tasks_are_scheduled_fairly {
use super::*;
struct Spin {
state: Rc<RefCell<[i32; 2]>>,
idx: usize,
}
impl Future for Spin {
type Item = ();
type Error = ();
fn poll(&mut self) -> Poll<(), ()> {
let mut state = self.state.borrow_mut();
if self.idx == 0 {
let diff = state[0] - state[1];
assert!(diff.abs() <= 1);
if state[0] >= 50 {
return Ok(().into());
}
}
state[self.idx] += 1;
if state[self.idx] >= 100 {
return Ok(().into());
}
task::current().notify();
Ok(Async::NotReady)
}
}
fn test<F: Fn(Spin)>(spawn: F) {
let state = Rc::new(RefCell::new([0, 0]));
block_on_all(lazy(|| {
spawn(Spin {
state: state.clone(),
idx: 0,
});
spawn(Spin {
state: state,
idx: 1,
});
ok()
}))
.unwrap();
}
#[test]
fn spawn() {
test(tokio_current_thread::spawn)
}
#[test]
fn execute() {
test(|f| {
tokio_current_thread::TaskExecutor::current()
.execute(f)
.unwrap();
})
}
}
mod and_turn {
use super::*;
fn test<F, G>(spawn: F, dotspawn: G)
where
F: Fn(Box<Future<Item = (), Error = ()>>) + 'static,
G: Fn(&mut CurrentThread, Box<Future<Item = (), Error = ()>>),
{
let cnt = Rc::new(Cell::new(0));
let c = cnt.clone();
let mut tokio_current_thread = CurrentThread::new();
// Spawn a basic task to get the executor to turn
dotspawn(&mut tokio_current_thread, Box::new(lazy(move || Ok(()))));
// Turn once...
tokio_current_thread.turn(None).unwrap();
dotspawn(
&mut tokio_current_thread,
Box::new(lazy(move || {
c.set(1 + c.get());
// Spawn!
spawn(Box::new(lazy(move || {
c.set(1 + c.get());
Ok::<(), ()>(())
})));
Ok(())
})),
);
// This does not run the newly spawned thread
tokio_current_thread.turn(None).unwrap();
assert_eq!(1, cnt.get());
// This runs the newly spawned thread
tokio_current_thread.turn(None).unwrap();
assert_eq!(2, cnt.get());
}
#[test]
fn spawn() {
test(tokio_current_thread::spawn, |rt, f| {
rt.spawn(f);
})
}
#[test]
fn execute() {
test(
|f| {
tokio_current_thread::TaskExecutor::current()
.execute(f)
.unwrap();
},
// Note: `CurrentThread` doesn't currently implement
// `futures::Executor`, so we'll call `.spawn(...)` rather than
// `.execute(...)` for now. If `CurrentThread` is changed to
// implement Executor, change this to `.execute(...).unwrap()`.
|rt, f| {
rt.spawn(f);
},
);
}
}
mod in_drop {
use super::*;
struct OnDrop<F: FnOnce()>(Option<F>);
impl<F: FnOnce()> Drop for OnDrop<F> {
fn drop(&mut self) {
(self.0.take().unwrap())();
}
}
struct MyFuture {
_data: Box<Any>,
}
impl Future for MyFuture {
type Item = ();
type Error = ();
fn poll(&mut self) -> Poll<(), ()> {
Ok(().into())
}
}
fn test<F, G>(spawn: F, dotspawn: G)
where
F: Fn(Box<Future<Item = (), Error = ()>>) + 'static,
G: Fn(&mut CurrentThread, Box<Future<Item = (), Error = ()>>),
{
let mut tokio_current_thread = CurrentThread::new();
let (tx, rx) = oneshot::channel();
dotspawn(
&mut tokio_current_thread,
Box::new(MyFuture {
_data: Box::new(OnDrop(Some(move || {
spawn(Box::new(lazy(move || {
tx.send(()).unwrap();
Ok(())
})));
}))),
}),
);
tokio_current_thread.block_on(rx).unwrap();
tokio_current_thread.run().unwrap();
}
#[test]
fn spawn() {
test(tokio_current_thread::spawn, |rt, f| {
rt.spawn(f);
})
}
#[test]
fn execute() {
test(
|f| {
tokio_current_thread::TaskExecutor::current()
.execute(f)
.unwrap();
},
// Note: `CurrentThread` doesn't currently implement
// `futures::Executor`, so we'll call `.spawn(...)` rather than
// `.execute(...)` for now. If `CurrentThread` is changed to
// implement Executor, change this to `.execute(...).unwrap()`.
|rt, f| {
rt.spawn(f);
},
);
}
}
#[test]
fn hammer_turn() {
use futures::sync::mpsc;
const ITER: usize = 100;
const N: usize = 100;
const THREADS: usize = 4;
for _ in 0..ITER {
let mut ths = vec![];
// Add some jitter
for _ in 0..THREADS {
let th = thread::spawn(|| {
let mut tokio_current_thread = CurrentThread::new();
let (tx, rx) = mpsc::unbounded();
tokio_current_thread.spawn({
let cnt = Rc::new(Cell::new(0));
let c = cnt.clone();
rx.for_each(move |_| {
c.set(1 + c.get());
Ok(())
})
.map_err(|e| panic!("err={:?}", e))
.map(move |v| {
assert_eq!(N, cnt.get());
v
})
});
thread::spawn(move || {
for _ in 0..N {
tx.unbounded_send(()).unwrap();
thread::yield_now();
}
});
while !tokio_current_thread.is_idle() {
tokio_current_thread.turn(None).unwrap();
}
});
ths.push(th);
}
for th in ths {
th.join().unwrap();
}
}
}
#[test]
fn turn_has_polled() {
let mut tokio_current_thread = CurrentThread::new();
// Spawn oneshot receiver
let (sender, receiver) = oneshot::channel::<()>();
tokio_current_thread.spawn(receiver.then(|_| Ok(())));
// Turn once...
let res = tokio_current_thread
.turn(Some(Duration::from_millis(0)))
.unwrap();
// Should've polled the receiver once, but considered it not ready
assert!(res.has_polled());
// Turn another time
let res = tokio_current_thread
.turn(Some(Duration::from_millis(0)))
.unwrap();
// Should've polled nothing, the receiver is not ready yet
assert!(!res.has_polled());
// Make the receiver ready
sender.send(()).unwrap();
// Turn another time
let res = tokio_current_thread
.turn(Some(Duration::from_millis(0)))
.unwrap();
// Should've polled the receiver, it's ready now
assert!(res.has_polled());
// Now the executor should be empty
assert!(tokio_current_thread.is_idle());
let res = tokio_current_thread
.turn(Some(Duration::from_millis(0)))
.unwrap();
// So should've polled nothing
assert!(!res.has_polled());
}
// Our own mock Park that is never really waiting and the only
// thing it does is to send, on request, something (once) to a oneshot
// channel
struct MyPark {
sender: Option<oneshot::Sender<()>>,
send_now: Rc<Cell<bool>>,
}
struct MyUnpark;
impl tokio_executor::park::Park for MyPark {
type Unpark = MyUnpark;
type Error = ();
fn unpark(&self) -> Self::Unpark {
MyUnpark
}
fn park(&mut self) -> Result<(), Self::Error> {
// If called twice with send_now, this will intentionally panic
if self.send_now.get() {
self.sender.take().unwrap().send(()).unwrap();
}
Ok(())
}
fn park_timeout(&mut self, _duration: Duration) -> Result<(), Self::Error> {
self.park()
}
}
impl tokio_executor::park::Unpark for MyUnpark {
fn unpark(&self) {}
}
#[test]
fn turn_fair() {
let send_now = Rc::new(Cell::new(false));
let (sender, receiver) = oneshot::channel::<()>();
let (sender_2, receiver_2) = oneshot::channel::<()>();
let (sender_3, receiver_3) = oneshot::channel::<()>();
let my_park = MyPark {
sender: Some(sender_3),
send_now: send_now.clone(),
};
let mut tokio_current_thread = CurrentThread::new_with_park(my_park);
let receiver_1_done = Rc::new(Cell::new(false));
let receiver_1_done_clone = receiver_1_done.clone();
// Once an item is received on the oneshot channel, it will immediately
// immediately make the second oneshot channel ready
tokio_current_thread.spawn(receiver.map_err(|_| unreachable!()).and_then(move |_| {
sender_2.send(()).unwrap();
receiver_1_done_clone.set(true);
Ok(())
}));
let receiver_2_done = Rc::new(Cell::new(false));
let receiver_2_done_clone = receiver_2_done.clone();
tokio_current_thread.spawn(receiver_2.map_err(|_| unreachable!()).and_then(move |_| {
receiver_2_done_clone.set(true);
Ok(())
}));
// The third receiver is only woken up from our Park implementation, it simulates
// e.g. a socket that first has to be polled to know if it is ready now
let receiver_3_done = Rc::new(Cell::new(false));
let receiver_3_done_clone = receiver_3_done.clone();
tokio_current_thread.spawn(receiver_3.map_err(|_| unreachable!()).and_then(move |_| {
receiver_3_done_clone.set(true);
Ok(())
}));
// First turn should've polled both and considered them not ready
let res = tokio_current_thread
.turn(Some(Duration::from_millis(0)))
.unwrap();
assert!(res.has_polled());
// Next turn should've polled nothing
let res = tokio_current_thread
.turn(Some(Duration::from_millis(0)))
.unwrap();
assert!(!res.has_polled());
assert!(!receiver_1_done.get());
assert!(!receiver_2_done.get());
assert!(!receiver_3_done.get());
// After this the receiver future will wake up the second receiver future,
// so there are pending futures again
sender.send(()).unwrap();
// Now the first receiver should be done, the second receiver should be ready
// to be polled again and the socket not yet
let res = tokio_current_thread.turn(None).unwrap();
assert!(res.has_polled());
assert!(receiver_1_done.get());
assert!(!receiver_2_done.get());
assert!(!receiver_3_done.get());
// Now let our park implementation know that it should send something to sender 3
send_now.set(true);
// This should resolve the second receiver directly, but also poll the socket
// and read the packet from it. If it didn't do both here, we would handle
// futures that are woken up from the reactor and directly unfairly and would
// favour the ones that are woken up directly.
let res = tokio_current_thread.turn(None).unwrap();
assert!(res.has_polled());
assert!(receiver_1_done.get());
assert!(receiver_2_done.get());
assert!(receiver_3_done.get());
// Don't send again
send_now.set(false);
// Now we should be idle and turning should not poll anything
assert!(tokio_current_thread.is_idle());
let res = tokio_current_thread.turn(None).unwrap();
assert!(!res.has_polled());
}
#[test]
fn spawn_from_other_thread() {
let mut current_thread = CurrentThread::new();
let handle = current_thread.handle();
let (sender, receiver) = oneshot::channel::<()>();
thread::spawn(move || {
handle
.spawn(lazy(move || {
sender.send(()).unwrap();
Ok(())
}))
.unwrap();
});
let _ = current_thread.block_on(receiver).unwrap();
}
#[test]
fn spawn_from_other_thread_unpark() {
use std::sync::mpsc::channel as mpsc_channel;
let mut current_thread = CurrentThread::new();
let handle = current_thread.handle();
let (sender_1, receiver_1) = oneshot::channel::<()>();
let (sender_2, receiver_2) = mpsc_channel::<()>();
thread::spawn(move || {
let _ = receiver_2.recv().unwrap();
handle
.spawn(lazy(move || {
sender_1.send(()).unwrap();
Ok(())
}))
.unwrap();
});
// Ensure that unparking the executor works correctly. It will first
// check if there are new futures (there are none), then execute the
// lazy future below which will cause the future to be spawned from
// the other thread. Then the executor will park but should be woken
// up because *now* we have a new future to schedule
let _ = current_thread
.block_on(
lazy(move || {
sender_2.send(()).unwrap();
Ok(())
})
.and_then(|_| receiver_1),
)
.unwrap();
}
#[test]
fn spawn_from_executor_with_handle() {
let mut current_thread = CurrentThread::new();
let handle = current_thread.handle();
let (tx, rx) = oneshot::channel();
current_thread.spawn(lazy(move || {
handle
.spawn(lazy(move || {
tx.send(()).unwrap();
Ok(())
}))
.unwrap();
Ok::<_, ()>(())
}));
current_thread.run();
rx.wait().unwrap();
}
fn ok() -> future::FutureResult<(), ()> {
future::ok(())
}
-38
View File
@@ -1,38 +0,0 @@
# 0.1.7 (March 22, 2019)
### Added
- `TypedExecutor` for spawning futures of a specific type (#993).
# 0.1.6 (January 6, 2019)
* Implement `Unpark` for `Arc<Unpark>` (#802).
* Switch to crossbeam's Parker / Unparker (#528).
# 0.1.5 (September 26, 2018)
* Implement `futures::Executor` for `DefaultExecutor` (#563).
* Add `Enter::block_on(future)` (#646)
# 0.1.4 (August 23, 2018)
* Implement `std::error::Error` for error types (#511).
# 0.1.3 (August 6, 2018)
* Implement `Executor` for `Box<E: Executor>` (#420).
* Improve `EnterError` debug message (#410).
* Implement `status`, `Send`, and `Sync` for `DefaultExecutor` (#463, #472).
* Fix race in `ParkThread` (#507).
* Handle recursive calls into `DefaultExecutor` (#473).
# 0.1.2 (March 30, 2018)
* Implement `Unpark` for `Box<Unpark>`.
# 0.1.1 (March 22, 2018)
* Optionally support futures 0.2.
# 0.1.0 (March 09, 2018)
* Initial release
-28
View File
@@ -1,28 +0,0 @@
[package]
name = "tokio-executor"
# When releasing to crates.io:
# - Remove path dependencies
# - Update html_root_url.
# - Update doc url
# - Cargo.toml
# - README.md
# - Update CHANGELOG.md.
# - Create "v0.1.x" git tag.
version = "0.1.7"
documentation = "https://docs.rs/tokio-executor/0.1.7/tokio_executor"
repository = "https://github.com/tokio-rs/tokio"
homepage = "https://github.com/tokio-rs/tokio"
license = "MIT"
authors = ["Carl Lerche <[email protected]>"]
description = """
Future execution primitives
"""
keywords = ["futures", "tokio"]
categories = ["concurrency", "asynchronous"]
[dependencies]
crossbeam-utils = "0.6.2"
futures = "0.1.19"
[dev-dependencies]
tokio = "0.1.18"
-25
View File
@@ -1,25 +0,0 @@
Copyright (c) 2019 Tokio Contributors
Permission is hereby granted, free of charge, to any
person obtaining a copy of this software and associated
documentation files (the "Software"), to deal in the
Software without restriction, including without
limitation the rights to use, copy, modify, merge,
publish, distribute, sublicense, and/or sell copies of
the Software, and to permit persons to whom the Software
is furnished to do so, subject to the following
conditions:
The above copyright notice and this permission notice
shall be included in all copies or substantial portions
of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF
ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED
TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A
PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT
SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY
CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR
IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
DEALINGS IN THE SOFTWARE.
-47
View File
@@ -1,47 +0,0 @@
# tokio-executor
Task execution related traits and utilities.
[Documentation](https://docs.rs/tokio-executor/0.1.7/tokio_executor)
## Overview
In the Tokio execution model, futures are lazy. When a future is created, no
work is performed. In order for the work defined by the future to happen, the
future must be submitted to an executor. A future that is submitted to an
executor is called a "task".
The executor is responsible for ensuring that [`Future::poll`] is called
whenever the task is [notified]. Notification happens when the internal state of
a task transitions from "not ready" to ready. For example, a socket might have
received data and a call to `read` will now be able to succeed.
This crate provides traits and utilities that are necessary for building an
executor, including:
* The [`Executor`] trait describes the API for spawning a future onto an
executor.
* [`enter`] marks that the current thread is entering an execution
context. This prevents a second executor from accidentally starting from
within the context of one that is already running.
* [`DefaultExecutor`] spawns tasks onto the default executor for the current
context.
* [`Park`] abstracts over blocking and unblocking the current thread.
[`Executor`]: https://docs.rs/tokio-executor/0.1.7/tokio_executor/trait.Executor.html
[`enter`]: https://docs.rs/tokio-executor/0.1.7/tokio_executor/fn.enter.html
[`DefaultExecutor`]: https://docs.rs/tokio-executor/0.1.7/tokio_executor/struct.DefaultExecutor.html
[`Park`]: https://docs.rs/tokio-executor/0.1.7/tokio_executor/park/trait.Park.html
## License
This project is licensed under the [MIT license](LICENSE).
### Contribution
Unless you explicitly state otherwise, any contribution intentionally submitted
for inclusion in Tokio by you, shall be licensed as MIT, without any additional
terms or conditions.
-128
View File
@@ -1,128 +0,0 @@
use std::cell::Cell;
use std::error::Error;
use std::fmt;
use std::prelude::v1::*;
use futures::{self, Future};
thread_local!(static ENTERED: Cell<bool> = Cell::new(false));
/// Represents an executor context.
///
/// For more details, see [`enter` documentation](fn.enter.html)
pub struct Enter {
on_exit: Vec<Box<Callback>>,
permanent: bool,
}
/// An error returned by `enter` if an execution scope has already been
/// entered.
pub struct EnterError {
_a: (),
}
impl fmt::Debug for EnterError {
fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
f.debug_struct("EnterError")
.field("reason", &self.description())
.finish()
}
}
impl fmt::Display for EnterError {
fn fmt(&self, fmt: &mut fmt::Formatter) -> fmt::Result {
write!(fmt, "{}", self.description())
}
}
impl Error for EnterError {
fn description(&self) -> &str {
"attempted to run an executor while another executor is already running"
}
}
/// Marks the current thread as being within the dynamic extent of an
/// executor.
///
/// Executor implementations should call this function before blocking the
/// thread. If `None` is returned, the executor should fail by panicking or
/// taking some other action without blocking the current thread. This prevents
/// deadlocks due to multiple executors competing for the same thread.
///
/// # Error
///
/// Returns an error if the current thread is already marked
pub fn enter() -> Result<Enter, EnterError> {
ENTERED.with(|c| {
if c.get() {
Err(EnterError { _a: () })
} else {
c.set(true);
Ok(Enter {
on_exit: Vec::new(),
permanent: false,
})
}
})
}
impl Enter {
/// Register a callback to be invoked if and when the thread
/// ceased to act as an executor.
pub fn on_exit<F>(&mut self, f: F)
where
F: FnOnce() + 'static,
{
self.on_exit.push(Box::new(f));
}
/// Treat the remainder of execution on this thread as part of an
/// executor; used mostly for thread pool worker threads.
///
/// All registered `on_exit` callbacks are *dropped* without being
/// invoked.
pub fn make_permanent(mut self) {
self.permanent = true;
}
/// Blocks the thread on the specified future, returning the value with
/// which that future completes.
pub fn block_on<F: Future>(&mut self, f: F) -> Result<F::Item, F::Error> {
futures::executor::spawn(f).wait_future()
}
}
impl fmt::Debug for Enter {
fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
f.debug_struct("Enter").finish()
}
}
impl Drop for Enter {
fn drop(&mut self) {
ENTERED.with(|c| {
assert!(c.get());
if self.permanent {
return;
}
for callback in self.on_exit.drain(..) {
callback.call();
}
c.set(false);
});
}
}
trait Callback: 'static {
fn call(self: Box<Self>);
}
impl<F: FnOnce() + 'static> Callback for F {
fn call(self: Box<Self>) {
(*self)()
}
}
-50
View File
@@ -1,50 +0,0 @@
use std::error::Error;
use std::fmt;
/// Errors returned by `Executor::spawn`.
///
/// Spawn errors should represent relatively rare scenarios. Currently, the two
/// scenarios represented by `SpawnError` are:
///
/// * An executor being at capacity or full. As such, the executor is not able
/// to accept a new future. This error state is expected to be transient.
/// * An executor has been shutdown and can no longer accept new futures. This
/// error state is expected to be permanent.
#[derive(Debug)]
pub struct SpawnError {
is_shutdown: bool,
}
impl SpawnError {
/// Return a new `SpawnError` reflecting a shutdown executor failure.
pub fn shutdown() -> Self {
SpawnError { is_shutdown: true }
}
/// Return a new `SpawnError` reflecting an executor at capacity failure.
pub fn at_capacity() -> Self {
SpawnError { is_shutdown: false }
}
/// Returns `true` if the error reflects a shutdown executor failure.
pub fn is_shutdown(&self) -> bool {
self.is_shutdown
}
/// Returns `true` if the error reflects an executor at capacity failure.
pub fn is_at_capacity(&self) -> bool {
!self.is_shutdown
}
}
impl fmt::Display for SpawnError {
fn fmt(&self, fmt: &mut fmt::Formatter) -> fmt::Result {
write!(fmt, "{}", self.description())
}
}
impl Error for SpawnError {
fn description(&self) -> &str {
"attempted to spawn task while the executor is at capacity or shut down"
}
}
-151
View File
@@ -1,151 +0,0 @@
use futures::Future;
use SpawnError;
/// A value that executes futures.
///
/// The [`spawn`] function is used to submit a future to an executor. Once
/// submitted, the executor takes ownership of the future and becomes
/// responsible for driving the future to completion.
///
/// The strategy employed by the executor to handle the future is less defined
/// and is left up to the `Executor` implementation. The `Executor` instance is
/// expected to call [`poll`] on the future once it has been notified, however
/// the "when" and "how" can vary greatly.
///
/// For example, the executor might be a thread pool, in which case a set of
/// threads have already been spawned up and the future is inserted into a
/// queue. A thread will acquire the future and poll it.
///
/// The `Executor` trait is only for futures that **are** `Send`. These are most
/// common. There currently is no trait that describes executors that operate
/// entirely on the current thread (i.e., are able to spawn futures that are not
/// `Send`). Note that single threaded executors can still implement `Executor`,
/// but only futures that are `Send` can be spawned via the trait.
///
/// This trait is primarily intended to implemented by executors and used to
/// back `tokio::spawn`. Libraries and applications **may** use this trait to
/// bound generics, but doing so will limit usage to futures that implement
/// `Send`. Instead, libraries and applications are recommended to use
/// [`TypedExecutor`] as a bound.
///
/// # Errors
///
/// The [`spawn`] function returns `Result` with an error type of `SpawnError`.
/// This error type represents the reason that the executor was unable to spawn
/// the future. The two current represented scenarios are:
///
/// * An executor being at capacity or full. As such, the executor is not able
/// to accept a new future. This error state is expected to be transient.
/// * An executor has been shutdown and can no longer accept new futures. This
/// error state is expected to be permanent.
///
/// If a caller encounters an at capacity error, the caller should try to shed
/// load. This can be as simple as dropping the future that was spawned.
///
/// If the caller encounters a shutdown error, the caller should attempt to
/// gracefully shutdown.
///
/// # Examples
///
/// ```rust
/// # extern crate futures;
/// # extern crate tokio_executor;
/// # use tokio_executor::Executor;
/// # fn docs(my_executor: &mut Executor) {
/// use futures::future::lazy;
/// my_executor.spawn(Box::new(lazy(|| {
/// println!("running on the executor");
/// Ok(())
/// }))).unwrap();
/// # }
/// # fn main() {}
/// ```
///
/// [`spawn`]: #tymethod.spawn
/// [`poll`]: https://docs.rs/futures/0.1/futures/future/trait.Future.html#tymethod.poll
/// [`TypedExecutor`]: ../trait.TypedExecutor.html
pub trait Executor {
/// Spawns a future object to run on this executor.
///
/// `future` is passed to the executor, which will begin running it. The
/// future may run on the current thread or another thread at the discretion
/// of the `Executor` implementation.
///
/// # Panics
///
/// Implementations are encouraged to avoid panics. However, panics are
/// permitted and the caller should check the implementation specific
/// documentation for more details on possible panics.
///
/// # Examples
///
/// ```rust
/// # extern crate futures;
/// # extern crate tokio_executor;
/// # use tokio_executor::Executor;
/// # fn docs(my_executor: &mut Executor) {
/// use futures::future::lazy;
/// my_executor.spawn(Box::new(lazy(|| {
/// println!("running on the executor");
/// Ok(())
/// }))).unwrap();
/// # }
/// # fn main() {}
/// ```
fn spawn(
&mut self,
future: Box<Future<Item = (), Error = ()> + Send>,
) -> Result<(), SpawnError>;
/// Provides a best effort **hint** to whether or not `spawn` will succeed.
///
/// This function may return both false positives **and** false negatives.
/// If `status` returns `Ok`, then a call to `spawn` will *probably*
/// succeed, but may fail. If `status` returns `Err`, a call to `spawn` will
/// *probably* fail, but may succeed.
///
/// This allows a caller to avoid creating the task if the call to `spawn`
/// has a high likelihood of failing.
///
/// # Panics
///
/// This function must not panic. Implementers must ensure that panics do
/// not happen.
///
/// # Examples
///
/// ```rust
/// # extern crate futures;
/// # extern crate tokio_executor;
/// # use tokio_executor::Executor;
/// # fn docs(my_executor: &mut Executor) {
/// use futures::future::lazy;
///
/// if my_executor.status().is_ok() {
/// my_executor.spawn(Box::new(lazy(|| {
/// println!("running on the executor");
/// Ok(())
/// }))).unwrap();
/// } else {
/// println!("the executor is not in a good state");
/// }
/// # }
/// # fn main() {}
/// ```
fn status(&self) -> Result<(), SpawnError> {
Ok(())
}
}
impl<E: Executor + ?Sized> Executor for Box<E> {
fn spawn(
&mut self,
future: Box<Future<Item = (), Error = ()> + Send>,
) -> Result<(), SpawnError> {
(**self).spawn(future)
}
fn status(&self) -> Result<(), SpawnError> {
(**self).status()
}
}
-240
View File
@@ -1,240 +0,0 @@
use super::{Enter, Executor, SpawnError};
use futures::{future, Future};
use std::cell::Cell;
/// Executes futures on the default executor for the current execution context.
///
/// `DefaultExecutor` implements `Executor` and can be used to spawn futures
/// without referencing a specific executor.
///
/// When an executor starts, it sets the `DefaultExecutor` handle to point to an
/// executor (usually itself) that is used to spawn new tasks.
///
/// The current `DefaultExecutor` reference is tracked using a thread-local
/// variable and is set using `tokio_executor::with_default`
#[derive(Debug, Clone)]
pub struct DefaultExecutor {
_dummy: (),
}
impl DefaultExecutor {
/// Returns a handle to the default executor for the current context.
///
/// Futures may be spawned onto the default executor using this handle.
///
/// The returned handle will reference whichever executor is configured as
/// the default **at the time `spawn` is called**. This enables
/// `DefaultExecutor::current()` to be called before an execution context is
/// setup, then passed **into** an execution context before it is used.
///
/// This is also true for sending the handle across threads, so calling
/// `DefaultExecutor::current()` on thread A and then sending the result to
/// thread B will _not_ reference the default executor that was set on thread A.
pub fn current() -> DefaultExecutor {
DefaultExecutor { _dummy: () }
}
#[inline]
fn with_current<F: FnOnce(&mut Executor) -> R, R>(f: F) -> Option<R> {
EXECUTOR.with(
|current_executor| match current_executor.replace(State::Active) {
State::Ready(executor_ptr) => {
let executor = unsafe { &mut *executor_ptr };
let result = f(executor);
current_executor.set(State::Ready(executor_ptr));
Some(result)
}
State::Empty | State::Active => None,
},
)
}
}
#[derive(Clone, Copy)]
enum State {
// default executor not defined
Empty,
// default executor is defined and ready to be used
Ready(*mut Executor),
// default executor is currently active (used to detect recursive calls)
Active,
}
thread_local! {
/// Thread-local tracking the current executor
static EXECUTOR: Cell<State> = Cell::new(State::Empty)
}
// ===== impl DefaultExecutor =====
impl super::Executor for DefaultExecutor {
fn spawn(
&mut self,
future: Box<Future<Item = (), Error = ()> + Send>,
) -> Result<(), SpawnError> {
DefaultExecutor::with_current(|executor| executor.spawn(future))
.unwrap_or_else(|| Err(SpawnError::shutdown()))
}
fn status(&self) -> Result<(), SpawnError> {
DefaultExecutor::with_current(|executor| executor.status())
.unwrap_or_else(|| Err(SpawnError::shutdown()))
}
}
impl<T> super::TypedExecutor<T> for DefaultExecutor
where
T: Future<Item = (), Error = ()> + Send + 'static,
{
fn spawn(&mut self, future: T) -> Result<(), SpawnError> {
super::Executor::spawn(self, Box::new(future))
}
fn status(&self) -> Result<(), SpawnError> {
super::Executor::status(self)
}
}
impl<T> future::Executor<T> for DefaultExecutor
where
T: Future<Item = (), Error = ()> + Send + 'static,
{
fn execute(&self, future: T) -> Result<(), future::ExecuteError<T>> {
if let Err(e) = super::Executor::status(self) {
let kind = if e.is_at_capacity() {
future::ExecuteErrorKind::NoCapacity
} else {
future::ExecuteErrorKind::Shutdown
};
return Err(future::ExecuteError::new(kind, future));
}
let _ = DefaultExecutor::with_current(|executor| executor.spawn(Box::new(future)));
Ok(())
}
}
// ===== global spawn fns =====
/// Submits a future for execution on the default executor -- usually a
/// threadpool.
///
/// Futures are lazy constructs. When they are defined, no work happens. In
/// order for the logic defined by the future to be run, the future must be
/// spawned on an executor. This function is the easiest way to do so.
///
/// This function must be called from an execution context, i.e. from a future
/// that has been already spawned onto an executor.
///
/// Once spawned, the future will execute. The details of how that happens is
/// left up to the executor instance. If the executor is a thread pool, the
/// future will be pushed onto a queue that a worker thread polls from. If the
/// executor is a "current thread" executor, the future might be polled
/// immediately from within the call to `spawn` or it might be pushed onto an
/// internal queue.
///
/// # Panics
///
/// This function will panic if the default executor is not set or if spawning
/// onto the default executor returns an error. To avoid the panic, use the
/// `DefaultExecutor` handle directly.
///
/// # Examples
///
/// ```rust
/// # extern crate futures;
/// # extern crate tokio_executor;
/// # use tokio_executor::spawn;
/// # pub fn dox() {
/// use futures::future::lazy;
///
/// spawn(lazy(|| {
/// println!("running on the default executor");
/// Ok(())
/// }));
/// # }
/// # pub fn main() {}
/// ```
pub fn spawn<T>(future: T)
where
T: Future<Item = (), Error = ()> + Send + 'static,
{
DefaultExecutor::current().spawn(Box::new(future)).unwrap()
}
/// Set the default executor for the duration of the closure
///
/// # Panics
///
/// This function panics if there already is a default executor set.
pub fn with_default<T, F, R>(executor: &mut T, enter: &mut Enter, f: F) -> R
where
T: Executor,
F: FnOnce(&mut Enter) -> R,
{
EXECUTOR.with(|cell| {
match cell.get() {
State::Ready(_) | State::Active => {
panic!("default executor already set for execution context")
}
_ => {}
}
// Ensure that the executor is removed from the thread-local context
// when leaving the scope. This handles cases that involve panicking.
struct Reset<'a>(&'a Cell<State>);
impl<'a> Drop for Reset<'a> {
fn drop(&mut self) {
self.0.set(State::Empty);
}
}
let _reset = Reset(cell);
// While scary, this is safe. The function takes a
// `&mut Executor`, which guarantees that the reference lives for the
// duration of `with_default`.
//
// Because we are always clearing the TLS value at the end of the
// function, we can cast the reference to 'static which thread-local
// cells require.
let executor = unsafe { hide_lt(executor as &mut _ as *mut _) };
cell.set(State::Ready(executor));
f(enter)
})
}
unsafe fn hide_lt<'a>(p: *mut (Executor + 'a)) -> *mut (Executor + 'static) {
use std::mem;
mem::transmute(p)
}
#[cfg(test)]
mod tests {
use super::{with_default, DefaultExecutor, Executor};
#[test]
fn default_executor_is_send_and_sync() {
fn assert_send_sync<T: Send + Sync>() {}
assert_send_sync::<DefaultExecutor>();
}
#[test]
fn nested_default_executor_status() {
let mut enter = super::super::enter().unwrap();
let mut executor = DefaultExecutor::current();
let result = with_default(&mut executor, &mut enter, |_| {
DefaultExecutor::current().status()
});
assert!(result.err().unwrap().is_shutdown())
}
}

Some files were not shown because too many files have changed in this diff Show More