Tracing

Use built-in workflow traces and custom spans to inspect job execution, task duration, and bottlenecks.

Tilebox traces workflow jobs automatically. Job submission creates a root trace, runners continue that trace across machines, and every task execution creates a span.

Built-in traces connect task order, dependencies, parallel execution, task duration, task status, runner identity, service identity, and logs emitted while a span was active.

Add custom spans

Use context.tracer inside a task to add spans around meaningful parts of your own code.

Python
Go

from tilebox.workflows import ExecutionContext, Task

class ProcessScene(Task):
    scene_id: str

    def execute(self, context: ExecutionContext) -> None:
        with context.tracer.span("download-scene") as span:
            span.set_attribute("scene_id", self.scene_id)
            # download input data

        with context.tracer.span("compute-index"):
            # perform expensive computation
            pass

package tasks

import (
	"context"

	"github.com/tilebox/tilebox-go/workflows/v1"
)

type ProcessScene struct{}

func (t *ProcessScene) Execute(ctx context.Context) error {
	return workflows.WithSpan(ctx, "compute-index", func(ctx context.Context) error {
		// perform expensive computation
		return nil
	})
}

Custom spans are nested under the current task span. Logs emitted inside the span are correlated with its trace_id and span_id.

Span status and exceptions

If a task raises an exception, Tilebox records the exception on the task span and marks the span as failed before the task is retried or marked failed.

For finer-grained error reporting, record errors on your custom spans before re-raising them.

Python
Go

class ProcessScene(Task):
    scene_id: str

    def execute(self, context: ExecutionContext) -> None:
        with context.tracer.span("publish-output") as span:
            try:
                # publish output
                pass
            except Exception as error:
                span.record_exception(error)
                raise

package tasks

import (
	"context"
	"fmt"

	"github.com/tilebox/tilebox-go/workflows/v1"
)

type ProcessScene struct{}

func (t *ProcessScene) Execute(ctx context.Context) error {
	return workflows.WithSpan(ctx, "publish-output", func(ctx context.Context) error {
		if err := publishOutput(); err != nil {
			return fmt.Errorf("failed to publish output: %w", err)
		}
		return nil
	})
}

func publishOutput() error {
	return nil
}

Query spans

You can retrieve spans for a job through the jobs client. Python results can also be converted to a pandas DataFrame.

Python
Go

from tilebox.workflows import Client

client = Client()
job = client.jobs().submit("process-scene", ProcessScene(scene_id="S2A_001"))

spans = client.jobs().query_spans(job)
for span in spans:
    print(span.name, span.status_code, span.duration)

df = spans.to_pandas()

package main

import (
	"context"
	"fmt"
	"log/slog"
	"time"

	"github.com/google/uuid"
	"github.com/tilebox/tilebox-go/workflows/v1"
)

func main() {
	ctx := context.Background()
	client := workflows.NewClient()
	jobID := uuid.MustParse("019e07b1-916b-0630-f3ba-f1c33235d174")

	for span, err := range client.Jobs.QuerySpans(ctx, jobID) {
		if err != nil {
			slog.ErrorContext(ctx, "failed to query job spans", slog.Any("error", err))
			return
		}

		fmt.Printf("%s %-40s %s\n",
			span.StartTime.Format(time.RFC3339Nano),
			span.Name,
			span.Duration(),
		)
	}
}

See Query telemetry for the log and span query APIs.

Export to another backend

Tilebox stores traces by default. To export spans to your own observability backend as well, configure an OpenTelemetry or Axiom integration when the runner process starts.