Skip to content

Debug a failed workflow run

Inspect task state, logs, traces, runner context, and cluster alignment when a workflow job fails or stays queued.

Use this guide when a workflow job fails, runs slower than expected, or stays queued. Tilebox records job state, task state, logs, traces, and runner context so you can identify whether the problem is in task code, task routing, dependencies, or the runner environment.

Open the job in the Tilebox Console, or use the Tilebox command-line tool if you already have the job ID.

Terminal window
tilebox job logs <job-id>
tilebox job spans <job-id>

Start with the task graph. A failed task often points to task code or runtime dependencies. A queued task often points to cluster, runner, or task registration mismatch.

Common checks:

  • The job was submitted to the intended cluster.
  • A runner is connected to the same cluster.
  • The runner advertises the submitted task identifier and compatible version.
  • Any task dependencies are complete.
  • Retry limits have not been exhausted.

Logs show messages emitted by task code and runner context attached by Tilebox.

Terminal window
tilebox job logs <job-id>

Use structured log fields in your tasks so the relevant scene ID, product ID, path, or model name appears in the log record.

Traces show task timing, parent-child relationships, custom spans, and failures.

Terminal window
tilebox job spans <job-id>

Use traces to find slow subtasks, repeated retries, and failures inside a specific custom span.

For direct runners, fix the code and restart the runner process. For release runners, publish a fixed release and deploy it.

If the fix is compatible with the failed task input schema and task major version, retry the job. If the change is breaking, submit a new job with a new task version.