Generators¶
The body of a with logfire.span
statement or a function decorated with @logfire.instrument
should not contain the yield
keyword, except in functions decorated with @contextlib.contextmanager
or @contextlib.asynccontextmanager
. To see the problem, consider this example:
import logfire
logfire.configure()
def generate_items():
with logfire.span('Generating items'):
for i in range(3):
yield i
# Or equivalently:
@logfire.instrument('Generating items')
def generate_items():
for i in range(3):
yield i
def main():
items = generate_items()
for item in items:
logfire.info(f'Got item {item}')
# break
logfire.info('After processing items')
main()
If you run this, everything seems fine:
The Got item
log lines are inside the Generating items
span, and the After processing items
log is outside it, as expected.
But if you uncomment the break
line, you'll see that the After processing items
log line is also inside the Generating items
span:
This is because the generate_items
generator is left suspended at the yield
statement, and the with logfire.span('Generating items'):
block is still active, so the After processing items
log sees that span as its parent. This is confusing, and can happen anytime that iteration over a generator is interrupted, including by exceptions.
If you run the same code with async generators:
import asyncio
import logfire
logfire.configure()
async def generate_items():
with logfire.span('Generating items'):
for i in range(3):
yield i
async def main():
items = generate_items()
async for item in items:
logfire.info(f'Got item {item}')
break
logfire.info('After processing items')
asyncio.run(main())
You'll see the same problem, as well as an exception like this in the logs:
Failed to detach context
Traceback (most recent call last):
File "async_generator_example.py", line 11, in generate_items
yield i
asyncio.exceptions.CancelledError
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "opentelemetry/context/__init__.py", line 154, in detach
_RUNTIME_CONTEXT.detach(token)
File "opentelemetry/context/contextvars_context.py", line 50, in detach
self._current_context.reset(token)
ValueError: <Token var=<ContextVar name='current_context' default={} at 0x10afa3f60> at 0x10de034c0> was created in a different Context
This is why generator functions are not traced by logfire.install_auto_tracing()
.
What you can do¶
Move the span outside the generator¶
If you're looping over a generator, wrapping the loop in a span is safe, e.g:
import logfire
logfire.configure()
def generate_items():
for i in range(3):
yield i
def main():
items = generate_items()
with logfire.span('Generating items'):
for item in items:
logfire.info(f'Got item {item}')
break
logfire.info('After processing items')
main()
This is fine because the with logfire.span
block doesn't contain the yield
directly in its body.
Use a generator as a context manager¶
yield
is OK when used to implement a context manager, e.g:
from contextlib import contextmanager
import logfire
logfire.configure()
@contextmanager
def my_context():
with logfire.span('Context manager span'):
yield
try:
with my_context():
logfire.info('Inside context manager')
raise ValueError()
except Exception:
logfire.exception('Error!')
logfire.info('After context manager')
This is fine because even if there's an exception inside the context manager, the with
statement will ensure that the my_context
generator is promptly closed, and the span will be closed with it. This is in contrast to using a generator as an iterator, where the loop can be interrupted more easily.
Create a context manager that closes the generator¶
with closing(generator)
can be used to ensure that the generator and thus the span within is closed even if the loop is interrupted, e.g:
from contextlib import closing
import logfire
logfire.configure()
def generate_items():
with logfire.span('Generating items'):
for i in range(3):
yield i
def main():
with closing(generate_items()) as items:
for item in items:
logfire.info(f'Got item {item}')
break
logfire.info('After processing items')
main()
However this means that users of generate_items
must always remember to use with closing
. To ensure that they have no choice but to do so, you can make generate_items
a context manager itself:
from contextlib import closing, contextmanager
import logfire
logfire.configure()
@contextmanager
def generate_items():
def generator():
with logfire.span('Generating items'):
for i in range(3):
yield i
with closing(generator()) as items:
yield items
def main():
with generate_items() as items:
for item in items:
logfire.info(f'Got item {item}')
break
logfire.info('After processing items')
main()