# Tracing

Tilebox traces workflow jobs automatically. Job submission creates a root trace, runners continue that trace across machines, and every task execution creates a span.

![Job Execution Trace View](/docs/assets/console/job-execution-light.png)

![Job Execution Trace View](/docs/assets/console/job-execution-dark.png)

Built-in traces connect task order, dependencies, parallel execution, task duration, task status, runner identity, service identity, and logs emitted while a span was active.

## Add custom spans

Use `context.tracer` inside a task to add spans around meaningful parts of your own code.

**Python**

```python title="Python"
from tilebox.workflows import ExecutionContext, Task

class ProcessScene(Task):
    scene_id: str

    def execute(self, context: ExecutionContext) -> None:
        with context.tracer.span("download-scene") as span:
            span.set_attribute("scene_id", self.scene_id)
            # download input data

        with context.tracer.span("compute-index"):
            # perform expensive computation
            pass
```

**Go**

```go title="Go"
package tasks

import (
	"context"

	"github.com/tilebox/tilebox-go/workflows/v1"
)

type ProcessScene struct{}

func (t *ProcessScene) Execute(ctx context.Context) error {
	return workflows.WithSpan(ctx, "compute-index", func(ctx context.Context) error {
		// perform expensive computation
		return nil
	})
}
```

Custom spans are nested under the current task span. Logs emitted inside the span are correlated with its `trace_id` and `span_id`.

## Span status and exceptions

If a task raises an exception, Tilebox records the exception on the task span and marks the span as failed before the task is retried or marked failed.

For finer-grained error reporting, record errors on your custom spans before re-raising them.

**Python**

```python title="Python"
class ProcessScene(Task):
    scene_id: str

    def execute(self, context: ExecutionContext) -> None:
        with context.tracer.span("publish-output") as span:
            try:
                # publish output
                pass
            except Exception as error:
                span.record_exception(error)
                raise
```

**Go**

```go title="Go"
package tasks

import (
	"context"
	"fmt"

	"github.com/tilebox/tilebox-go/workflows/v1"
)

type ProcessScene struct{}

func (t *ProcessScene) Execute(ctx context.Context) error {
	return workflows.WithSpan(ctx, "publish-output", func(ctx context.Context) error {
		if err := publishOutput(); err != nil {
			return fmt.Errorf("failed to publish output: %w", err)
		}
		return nil
	})
}

func publishOutput() error {
	return nil
}
```

## Query spans

You can retrieve spans for a job through the jobs client. Python results can also be converted to a pandas DataFrame.

**Python**

```python title="Python"
from tilebox.workflows import Client

client = Client()
job = client.jobs().submit("process-scene", ProcessScene(scene_id="S2A_001"))

spans = client.jobs().query_spans(job)
for span in spans:
    print(span.name, span.status_code, span.duration)

df = spans.to_pandas()
```

**Go**

```go title="Go"
package main

import (
	"context"
	"fmt"
	"log/slog"
	"time"

	"github.com/google/uuid"
	"github.com/tilebox/tilebox-go/workflows/v1"
)

func main() {
	ctx := context.Background()
	client := workflows.NewClient()
	jobID := uuid.MustParse("019e07b1-916b-0630-f3ba-f1c33235d174")

	for span, err := range client.Jobs.QuerySpans(ctx, jobID) {
		if err != nil {
			slog.ErrorContext(ctx, "failed to query job spans", slog.Any("error", err))
			return
		}

		fmt.Printf("%s %-40s %s\n",
			span.StartTime.Format(time.RFC3339Nano),
			span.Name,
			span.Duration(),
		)
	}
}
```

See [Query telemetry](/docs/workflows/run-and-inspect/query-telemetry) for the log and span query APIs.

## Export to another backend

Tilebox stores traces by default. To export spans to your own observability backend as well, configure an [OpenTelemetry](/docs/workflows/run-and-inspect/integrations/open-telemetry) or [Axiom](/docs/workflows/run-and-inspect/integrations/axiom) integration when the runner process starts.