Introduce tokio-trace (#827)

<!-- Thank you for your Pull Request. Please provide a description above
and review the requirements below.

Bug fixes and new features should include tests.

Contributors guide:
https://github.com/tokio-rs/tokio/blob/master/CONTRIBUTING.md -->

## Motivation

In asynchronous systems like Tokio, interpreting traditional log
messages can often be quite challenging. Since individual tasks are
multiplexed on the same thread, associated events and log lines are
intermixed making it difficult to trace the logic flow. Currently, none
of the available logging frameworks or libraries in Rust offer the
ability to trace logical paths through a futures-based program.

There also are complementary goals that can be accomplished with such a
system. For example, metrics / instrumentation can be tracked by
observing emitted events, or trace data can be exported to a distributed
tracing or event processing system.

In addition, it can often be useful to generate this diagnostic data in
a structured manner that can be consumed programmatically. While prior
art for structured logging in Rust exists, it is not currently
standardized, and is not "Tokio-friendly".

## Solution

This branch adds a new library to the tokio project, `tokio-trace`.
`tokio-trace` expands upon logging-style diagnostics by allowing
libraries and applications to record structured events with additional
information about *temporality* and *causality* --- unlike a log
message, a span in `tokio-trace` has a beginning and end time, may be
entered and exited by the flow of execution, and may exist within a
nested tree of similar spans. In addition, `tokio-trace` spans are
*structured*, with the ability to record typed data as well as textual
messages.

The `tokio-trace-core` crate contains the core primitives for this
system, which are expected to remain stable, while `tokio-trace` crate
provides a more "batteries-included" API. In particular, it provides
macros which are a superset of the `log` crate's `error!`, `warn!`,
`info!`, `debug!`, and `trace!` macros, allowing users to begin the
process of adopting `tokio-trace` by performing a drop-in replacement.

## Notes

Work on this project had previously been carried out in the
[tokio-trace-prototype] repository. In addition to the `tokio-trace` and
`tokio-trace-core` crates, the `tokio-trace-prototype` repo also
contains prototypes or sketches of adapter, compatibility, and utility
crates which provide useful functionality for `tokio-trace`, but these
crates are not yet ready for a release. When this branch is merged, that
repository will be archived, and the remaining unstable crates will be
moved to a new `tokio-trace-nursery` repository. Remaining issues on the
`tokio-trace-prototype` repo will be moved to the appropriate new repo.

The crates added in this branch are not _identical_ to the current head
of the `tokio-trace-prototype` repo, as I did some final clean-up and docs
polish in this branch prior to merging this PR.

[tokio-trace-prototype]: https://github.com/hawkw/tokio-trace-prototype

Closes: #561

Signed-off-by: Eliza Weisman <[email protected]>
This commit is contained in:
Eliza Weisman
2019-02-19 12:15:01 -08:00
committed by GitHub
parent d1d72dc1c8
commit c08e73c8d4
35 changed files with 6358 additions and 0 deletions
+92
View File
@@ -0,0 +1,92 @@
#![allow(missing_docs)]
use super::{field, metadata};
use std::fmt;
/// A mock event.
///
/// This is intended for use with the mock subscriber API in the
/// `subscriber` module.
#[derive(Debug, Default, Eq, PartialEq)]
pub struct MockEvent {
pub fields: Option<field::Expect>,
metadata: metadata::Expect,
}
pub fn mock() -> MockEvent {
MockEvent {
..Default::default()
}
}
impl MockEvent {
pub fn named<I>(self, name: I) -> Self
where
I: Into<String>,
{
Self {
metadata: metadata::Expect {
name: Some(name.into()),
..self.metadata
},
..self
}
}
pub fn with_fields<I>(self, fields: I) -> Self
where
I: Into<field::Expect>,
{
Self {
fields: Some(fields.into()),
..self
}
}
pub fn at_level(self, level: tokio_trace::Level) -> Self {
Self {
metadata: metadata::Expect {
level: Some(level),
..self.metadata
},
..self
}
}
pub fn with_target<I>(self, target: I) -> Self
where
I: Into<String>,
{
Self {
metadata: metadata::Expect {
target: Some(target.into()),
..self.metadata
},
..self
}
}
pub(in support) fn check(self, event: &tokio_trace::Event) {
let meta = event.metadata();
let name = meta.name();
self.metadata.check(meta, format_args!("event {}", name));
if let Some(mut expected_fields) = self.fields {
let mut checker = expected_fields.checker(format!("{}", name));
event.record(&mut checker);
checker.finish();
}
}
}
impl fmt::Display for MockEvent {
fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
write!(f, "an event")?;
if let Some(ref name) = self.metadata.name {
write!(f, " named {:?}", name)?;
}
if let Some(ref fields) = self.fields {
write!(f, " with {}", fields)?
}
Ok(())
}
}
+224
View File
@@ -0,0 +1,224 @@
use tokio_trace::{
callsite::Callsite,
field::{self, Field, Record, Value},
};
use std::{collections::HashMap, fmt};
#[derive(Default, Debug, Eq, PartialEq)]
pub struct Expect {
fields: HashMap<String, MockValue>,
only: bool,
}
#[derive(Debug)]
pub struct MockField {
name: String,
value: MockValue,
}
#[derive(Debug, Eq, PartialEq)]
pub enum MockValue {
I64(i64),
U64(u64),
Bool(bool),
Str(String),
Debug(String),
Any,
}
pub fn mock<K>(name: K) -> MockField
where
String: From<K>,
{
MockField {
name: name.into(),
value: MockValue::Any,
}
}
impl MockField {
/// Expect a field with the given name and value.
pub fn with_value(self, value: &Value) -> Self {
Self {
value: MockValue::from(value),
..self
}
}
pub fn and(self, other: MockField) -> Expect {
Expect {
fields: HashMap::new(),
only: false,
}
.and(self)
.and(other)
}
pub fn only(self) -> Expect {
Expect {
fields: HashMap::new(),
only: true,
}
.and(self)
}
}
impl Into<Expect> for MockField {
fn into(self) -> Expect {
Expect {
fields: HashMap::new(),
only: false,
}
.and(self)
}
}
impl Expect {
pub fn and(mut self, field: MockField) -> Self {
self.fields.insert(field.name, field.value);
self
}
/// Indicates that no fields other than those specified should be expected.
pub fn only(self) -> Self {
Self { only: true, ..self }
}
fn compare_or_panic(&mut self, name: &str, value: &Value, ctx: &str) {
let value = value.into();
match self.fields.remove(name) {
Some(MockValue::Any) => {}
Some(expected) => assert!(
expected == value,
"\nexpected `{}` to contain:\n\t`{}{}`\nbut got:\n\t`{}{}`",
ctx,
name,
expected,
name,
value
),
None if self.only => panic!(
"\nexpected `{}` to contain only:\n\t`{}`\nbut got:\n\t`{}{}`",
ctx, self, name, value
),
_ => {}
}
}
pub fn checker<'a>(&'a mut self, ctx: String) -> CheckRecorder<'a> {
CheckRecorder { expect: self, ctx }
}
pub fn is_empty(&self) -> bool {
self.fields.is_empty()
}
}
impl fmt::Display for MockValue {
fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
match self {
MockValue::I64(v) => write!(f, ": i64 = {:?}", v),
MockValue::U64(v) => write!(f, ": u64 = {:?}", v),
MockValue::Bool(v) => write!(f, ": bool = {:?}", v),
MockValue::Str(v) => write!(f, ": &str = {:?}", v),
MockValue::Debug(v) => write!(f, ": &fmt::Debug = {:?}", v),
MockValue::Any => write!(f, ": _ = _"),
}
}
}
pub struct CheckRecorder<'a> {
expect: &'a mut Expect,
ctx: String,
}
impl<'a> Record for CheckRecorder<'a> {
fn record_i64(&mut self, field: &Field, value: i64) {
self.expect
.compare_or_panic(field.name(), &value, &self.ctx[..])
}
fn record_u64(&mut self, field: &Field, value: u64) {
self.expect
.compare_or_panic(field.name(), &value, &self.ctx[..])
}
fn record_bool(&mut self, field: &Field, value: bool) {
self.expect
.compare_or_panic(field.name(), &value, &self.ctx[..])
}
fn record_str(&mut self, field: &Field, value: &str) {
self.expect
.compare_or_panic(field.name(), &value, &self.ctx[..])
}
fn record_debug(&mut self, field: &Field, value: &fmt::Debug) {
self.expect
.compare_or_panic(field.name(), &field::debug(value), &self.ctx)
}
}
impl<'a> CheckRecorder<'a> {
pub fn finish(self) {
assert!(
self.expect.fields.is_empty(),
"{}missing {}",
self.expect,
self.ctx
);
}
}
impl<'a> From<&'a Value> for MockValue {
fn from(value: &'a Value) -> Self {
struct MockValueBuilder {
value: Option<MockValue>,
}
impl Record for MockValueBuilder {
fn record_i64(&mut self, _: &Field, value: i64) {
self.value = Some(MockValue::I64(value));
}
fn record_u64(&mut self, _: &Field, value: u64) {
self.value = Some(MockValue::U64(value));
}
fn record_bool(&mut self, _: &Field, value: bool) {
self.value = Some(MockValue::Bool(value));
}
fn record_str(&mut self, _: &Field, value: &str) {
self.value = Some(MockValue::Str(value.to_owned()));
}
fn record_debug(&mut self, _: &Field, value: &fmt::Debug) {
self.value = Some(MockValue::Debug(format!("{:?}", value)));
}
}
let fake_field = callsite!(name: "fake", fields: fake_field)
.metadata()
.fields()
.field("fake_field")
.unwrap();
let mut builder = MockValueBuilder { value: None };
value.record(&fake_field, &mut builder);
builder
.value
.expect("finish called before a value was recorded")
}
}
impl fmt::Display for Expect {
fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
write!(f, "fields ")?;
let entries = self
.fields
.iter()
.map(|(k, v)| (field::display(k), field::display(v)));
f.debug_map().entries(entries).finish()
}
}
+64
View File
@@ -0,0 +1,64 @@
use std::fmt;
use tokio_trace::Metadata;
#[derive(Debug, Eq, PartialEq, Default)]
pub struct Expect {
pub name: Option<String>,
pub level: Option<tokio_trace::Level>,
pub target: Option<String>,
}
impl Expect {
pub(in support) fn check(&self, actual: &Metadata, ctx: fmt::Arguments) {
if let Some(ref expected_name) = self.name {
let name = actual.name();
assert!(
expected_name == name,
"expected {} to be named `{}`, but got one named `{}`",
ctx,
expected_name,
name
)
}
if let Some(ref expected_level) = self.level {
let level = actual.level();
assert!(
expected_level == level,
"expected {} to be at level `{:?}`, but it was at level `{:?}` instead",
ctx,
expected_level,
level,
)
}
if let Some(ref expected_target) = self.target {
let target = actual.target();
assert!(
expected_target == &target,
"expected {} to have target `{}`, but it had target `{}` instead",
ctx,
expected_target,
target,
)
}
}
}
impl fmt::Display for Expect {
fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
if let Some(ref name) = self.name {
write!(f, "named `{}`", name)?;
}
if let Some(ref level) = self.level {
write!(f, " at the `{:?}` level", level)?;
}
if let Some(ref target) = self.target {
write!(f, " with target `{}`", target)?;
}
Ok(())
}
}
+6
View File
@@ -0,0 +1,6 @@
#![allow(dead_code)]
pub mod event;
pub mod field;
mod metadata;
pub mod span;
pub mod subscriber;
+109
View File
@@ -0,0 +1,109 @@
#![allow(missing_docs)]
use super::{field, metadata};
use std::fmt;
/// A mock span.
///
/// This is intended for use with the mock subscriber API in the
/// `subscriber` module.
#[derive(Debug, Default, Eq, PartialEq)]
pub struct MockSpan {
pub(in support) metadata: metadata::Expect,
}
#[derive(Debug, Default, Eq, PartialEq)]
pub struct NewSpan {
pub(in support) span: MockSpan,
pub(in support) fields: field::Expect,
}
pub fn mock() -> MockSpan {
MockSpan {
..Default::default()
}
}
impl MockSpan {
pub fn named<I>(self, name: I) -> Self
where
I: Into<String>,
{
Self {
metadata: metadata::Expect {
name: Some(name.into()),
..self.metadata
},
..self
}
}
pub fn at_level(self, level: tokio_trace::Level) -> Self {
Self {
metadata: metadata::Expect {
level: Some(level),
..self.metadata
},
..self
}
}
pub fn with_target<I>(self, target: I) -> Self
where
I: Into<String>,
{
Self {
metadata: metadata::Expect {
target: Some(target.into()),
..self.metadata
},
..self
}
}
pub fn name(&self) -> Option<&str> {
self.metadata.name.as_ref().map(String::as_ref)
}
pub fn with_field<I>(self, fields: I) -> NewSpan
where
I: Into<field::Expect>,
{
NewSpan {
span: self,
fields: fields.into(),
}
}
pub(in support) fn check_metadata(&self, actual: &tokio_trace::Metadata) {
self.metadata.check(actual, format_args!("span {}", self))
}
}
impl fmt::Display for MockSpan {
fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
if self.metadata.name.is_some() {
write!(f, "a span{}", self.metadata)
} else {
write!(f, "any span{}", self.metadata)
}
}
}
impl Into<NewSpan> for MockSpan {
fn into(self) -> NewSpan {
NewSpan {
span: self,
..Default::default()
}
}
}
impl fmt::Display for NewSpan {
fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
write!(f, "a new span{}", self.span.metadata)?;
if !self.fields.is_empty() {
write!(f, " with {}", self.fields)?;
}
Ok(())
}
}
+337
View File
@@ -0,0 +1,337 @@
#![allow(missing_docs)]
use super::{
event::MockEvent,
field as mock_field,
span::{MockSpan, NewSpan},
};
use std::{
collections::{HashMap, VecDeque},
fmt,
sync::{
atomic::{AtomicUsize, Ordering},
Arc, Mutex,
},
};
use tokio_trace::{field, Event, Id, Metadata, Subscriber};
#[derive(Debug, Eq, PartialEq)]
enum Expect {
Event(MockEvent),
Enter(MockSpan),
Exit(MockSpan),
CloneSpan(MockSpan),
DropSpan(MockSpan),
Record(MockSpan, mock_field::Expect),
NewSpan(NewSpan),
Nothing,
}
struct SpanState {
name: &'static str,
refs: usize,
}
struct Running<F: Fn(&Metadata) -> bool> {
spans: Mutex<HashMap<Id, SpanState>>,
expected: Arc<Mutex<VecDeque<Expect>>>,
ids: AtomicUsize,
filter: F,
}
pub struct MockSubscriber<F: Fn(&Metadata) -> bool> {
expected: VecDeque<Expect>,
filter: F,
}
pub struct MockHandle(Arc<Mutex<VecDeque<Expect>>>);
pub fn mock() -> MockSubscriber<fn(&Metadata) -> bool> {
MockSubscriber {
expected: VecDeque::new(),
filter: (|_: &Metadata| true) as for<'r, 's> fn(&'r Metadata<'s>) -> _,
}
}
impl<F: Fn(&Metadata) -> bool> MockSubscriber<F> {
pub fn enter(mut self, span: MockSpan) -> Self {
self.expected.push_back(Expect::Enter(span));
self
}
pub fn event(mut self, event: MockEvent) -> Self {
self.expected.push_back(Expect::Event(event));
self
}
pub fn exit(mut self, span: MockSpan) -> Self {
self.expected.push_back(Expect::Exit(span));
self
}
pub fn clone_span(mut self, span: MockSpan) -> Self {
self.expected.push_back(Expect::CloneSpan(span));
self
}
pub fn drop_span(mut self, span: MockSpan) -> Self {
self.expected.push_back(Expect::DropSpan(span));
self
}
pub fn done(mut self) -> Self {
self.expected.push_back(Expect::Nothing);
self
}
pub fn record<I>(mut self, span: MockSpan, fields: I) -> Self
where
I: Into<mock_field::Expect>,
{
self.expected.push_back(Expect::Record(span, fields.into()));
self
}
pub fn new_span<I>(mut self, new_span: I) -> Self
where
I: Into<NewSpan>,
{
self.expected.push_back(Expect::NewSpan(new_span.into()));
self
}
pub fn with_filter<G>(self, filter: G) -> MockSubscriber<G>
where
G: Fn(&Metadata) -> bool,
{
MockSubscriber {
filter,
expected: self.expected,
}
}
pub fn run(self) -> impl Subscriber {
let (subscriber, _) = self.run_with_handle();
subscriber
}
pub fn run_with_handle(self) -> (impl Subscriber, MockHandle) {
let expected = Arc::new(Mutex::new(self.expected));
let handle = MockHandle(expected.clone());
let subscriber = Running {
spans: Mutex::new(HashMap::new()),
expected,
ids: AtomicUsize::new(0),
filter: self.filter,
};
(subscriber, handle)
}
}
impl<F: Fn(&Metadata) -> bool> Subscriber for Running<F> {
fn enabled(&self, meta: &Metadata) -> bool {
(self.filter)(meta)
}
fn record(&self, id: &Id, values: &field::ValueSet) {
let spans = self.spans.lock().unwrap();
let mut expected = self.expected.lock().unwrap();
let span = spans
.get(id)
.unwrap_or_else(|| panic!("no span for ID {:?}", id));
println!("record: {}; id={:?}; values={:?};", span.name, id, values);
let was_expected = if let Some(Expect::Record(_, _)) = expected.front() {
true
} else {
false
};
if was_expected {
if let Expect::Record(expected_span, mut expected_values) =
expected.pop_front().unwrap()
{
if let Some(name) = expected_span.name() {
assert_eq!(name, span.name);
}
let mut checker = expected_values.checker(format!("span {}: ", span.name));
values.record(&mut checker);
checker.finish();
}
}
}
fn event(&self, event: &Event) {
let name = event.metadata().name();
println!("event: {};", name);
match self.expected.lock().unwrap().pop_front() {
None => {}
Some(Expect::Event(expected)) => expected.check(event),
Some(ex) => ex.bad(format_args!("observed event {:?}", event)),
}
}
fn record_follows_from(&self, _span: &Id, _follows: &Id) {
// TODO: it should be possible to expect spans to follow from other spans
}
fn new_span(&self, meta: &Metadata, values: &field::ValueSet) -> Id {
let id = self.ids.fetch_add(1, Ordering::SeqCst);
let id = Id::from_u64(id as u64);
println!(
"new_span: name={:?}; target={:?}; id={:?};",
meta.name(),
meta.target(),
id
);
let mut expected = self.expected.lock().unwrap();
let was_expected = match expected.front() {
Some(Expect::NewSpan(_)) => true,
_ => false,
};
if was_expected {
if let Expect::NewSpan(mut expected) = expected.pop_front().unwrap() {
let name = meta.name();
expected
.span
.metadata
.check(meta, format_args!("span `{}`", name));
let mut checker = expected.fields.checker(format!("{}", name));
values.record(&mut checker);
checker.finish();
}
}
self.spans.lock().unwrap().insert(
id.clone(),
SpanState {
name: meta.name(),
refs: 1,
},
);
id
}
fn enter(&self, id: &Id) {
let spans = self.spans.lock().unwrap();
if let Some(span) = spans.get(id) {
println!("enter: {}; id={:?};", span.name, id);
match self.expected.lock().unwrap().pop_front() {
None => {}
Some(Expect::Enter(ref expected_span)) => {
if let Some(name) = expected_span.name() {
assert_eq!(name, span.name);
}
}
Some(ex) => ex.bad(format_args!("entered span {:?}", span.name)),
}
};
}
fn exit(&self, id: &Id) {
let spans = self.spans.lock().unwrap();
let span = spans
.get(id)
.unwrap_or_else(|| panic!("no span for ID {:?}", id));
println!("exit: {}; id={:?};", span.name, id);
match self.expected.lock().unwrap().pop_front() {
None => {}
Some(Expect::Exit(ref expected_span)) => {
if let Some(name) = expected_span.name() {
assert_eq!(name, span.name);
}
}
Some(ex) => ex.bad(format_args!("exited span {:?}", span.name)),
};
}
fn clone_span(&self, id: &Id) -> Id {
let name = self.spans.lock().unwrap().get_mut(id).map(|span| {
let name = span.name;
println!("clone_span: {}; id={:?}; refs={:?};", name, id, span.refs);
span.refs += 1;
name
});
if name.is_none() {
println!("clone_span: id={:?};", id);
}
let mut expected = self.expected.lock().unwrap();
let was_expected = if let Some(Expect::CloneSpan(ref span)) = expected.front() {
assert_eq!(name, span.name());
true
} else {
false
};
if was_expected {
expected.pop_front();
}
id.clone()
}
fn drop_span(&self, id: Id) {
let mut is_event = false;
let name = if let Ok(mut spans) = self.spans.try_lock() {
spans.get_mut(&id).map(|span| {
let name = span.name;
if name.contains("event") {
is_event = true;
}
println!("drop_span: {}; id={:?}; refs={:?};", name, id, span.refs);
span.refs -= 1;
name
})
} else {
None
};
if name.is_none() {
println!("drop_span: id={:?}", id);
}
if let Ok(mut expected) = self.expected.try_lock() {
let was_expected = match expected.front() {
Some(Expect::DropSpan(ref span)) => {
// Don't assert if this function was called while panicking,
// as failing the assertion can cause a double panic.
if !::std::thread::panicking() {
assert_eq!(name, span.name());
}
true
}
Some(Expect::Event(_)) => {
if !::std::thread::panicking() {
assert!(is_event);
}
true
}
_ => false,
};
if was_expected {
expected.pop_front();
}
}
}
}
impl MockHandle {
pub fn assert_finished(&self) {
if let Ok(ref expected) = self.0.lock() {
assert!(
!expected.iter().any(|thing| thing != &Expect::Nothing),
"more notifications expected: {:?}",
**expected
);
}
}
}
impl Expect {
fn bad<'a>(&self, what: fmt::Arguments<'a>) {
match self {
Expect::Event(e) => panic!("expected event {}, but {} instead", e, what,),
Expect::Enter(e) => panic!("expected to enter {} but {} instead", e, what,),
Expect::Exit(e) => panic!("expected to exit {} but {} instead", e, what,),
Expect::CloneSpan(e) => panic!("expected to clone {} but {} instead", e, what,),
Expect::DropSpan(e) => panic!("expected to drop {} but {} instead", e, what,),
Expect::Record(e, fields) => {
panic!("expected {} to record {} but {} instead", e, fields, what,)
}
Expect::NewSpan(e) => panic!("expected {} but {} instead", e, what),
Expect::Nothing => panic!("expected nothing else to happen, but {} instead", what,),
}
}
}