Multi-language Workflows

Open in GitHub

The code for this guide is available on GitHub.

Tilebox languages and SDKs

Tilebox supports multiple languages and SDKs for running workflows. All Tilebox SDKs and workflows are designed to be interoperable, which means it’s possible to have a workflow where individual tasks are executed in different languages. Check out Languages & SDKs to learn more about currently available programming SDKs.

Why multi-language workflows?

You might need to use multiple languages in a single workflow for many reasons, such as:

• You want to use a language that is better suited for a specific task (for example Python for data processing, Go for a backend API)
• You want to use a library that is only available in a specific language (for example xarray in Python)
• You started prototyping in Python, but need to start migrating the compute-intensive parts of your workflow to a different language for performance reasons

Multi-language workflow example

This guide will tackle the first use case: you have a tasking server in Go and want to offload some of the processing to Python.

Defining tasks in Python and Go

Python

class ScheduleImageCapture(Task):
    # The input parameters must match the ones defined in the Go task
    location: tuple[float, float]  # lat_lon
    resolution_m: int
    spectral_bands: list[float]  # spectral bands in nm

    def execute(self, context: ExecutionContext) -> None:
        # Here you can implement your task logic, submit subtasks, etc.
        context.logger.info(
            "Image captured",
            location=self.location,
            resolution_m=self.resolution_m,
            spectral_bands=self.spectral_bands,
        )

    @staticmethod
    def identifier() -> tuple[str, str]:
        # The identifier must match the one defined in the Go task
        return "tilebox.com/schedule_image_capture", "v1.0"

Copy

Go

type ScheduleImageCapture struct {
    // json tags must match the Python task definition
    Location      [2]float64 `json:"location"` // lat_lon
    ResolutionM   int        `json:"resolution_m"`
    SpectralBands []float64  `json:"spectral_bands"` // spectral bands in nm
}

// No need to define the Execute method since we're only submitting the task

// Identifier must match with the task identifier in the Python runner
func (t *ScheduleImageCapture) Identifier() workflows.TaskIdentifier {
    return workflows.NewTaskIdentifier("tilebox.com/schedule_image_capture", "v1.0")
}

Copy

A couple important points to note:

Input parameters

The dataclass parameters in Python must match the struct fields in Go, including the types and the names (through the JSON tags in Go).

Due to Go and Python having different naming conventions, it’s recommended to use JSON tags in the Go struct to match the Python dataclass field names to respect the language-specific conventions.

Go fields must start with an uppercase letter to be serialized to JSON.

The need for JSON tags in the preceding Go code is currently necessary but might be removed in the future.

Execute method

The execute method is defined in the Python task but not in the Go task since Go will only be used to submit the task, not executing it.

Identifier

It’s necessary to define the identifier method in both the Python and Go tasks and to make sure they match. The identifier method has two values, the first being an arbitrary unique task identifier and the second being the version in the v{major}.{minor} format.

Creating a Go server that submits jobs

Write a simple HTTP tasking server in Go with a /submit endpoint that accepts requests to submit a ScheduleImageCapture job.

Both Go and Python code are using test-cluster-tZD9Ca2qsqt4V as the cluster slug. You should replace it with your own cluster slug, which you can create in the Tilebox Console.

Go

func main() {
    log.Println("Server starting on http://localhost:8080")

    client := workflows.NewClient()
    http.HandleFunc("/submit", submitHandler(client))
    log.Fatal(http.ListenAndServe(":8080", nil))
}

// Submit a job based on some query parameters
func submitHandler(client *workflows.Client) http.HandlerFunc {
    return func(w http.ResponseWriter, r *http.Request) {
        latArg := r.URL.Query().Get("lat")
        lonArg := r.URL.Query().Get("lon")
        resolutionArg := r.URL.Query().Get("resolution")
        bandsArg := r.URL.Query().Get("bands[]")

        latFloat, err := strconv.ParseFloat(latArg, 64)
        if err != nil {
            http.Error(w, err.Error(), http.StatusBadRequest)
            return
        }
        lonFloat, err := strconv.ParseFloat(lonArg, 64)
        if err != nil {
            http.Error(w, err.Error(), http.StatusBadRequest)
            return
        }

        resolutionM, err := strconv.Atoi(resolutionArg)
        if err != nil {
            http.Error(w, err.Error(), http.StatusBadRequest)
            return
        }

        var spectralBands []float64
        for _, bandArg := range strings.Split(bandsArg, ",") {
            band, err := strconv.ParseFloat(bandArg, 64)
            if err != nil {
                http.Error(w, err.Error(), http.StatusBadRequest)
                return
            }
            spectralBands = append(spectralBands, band)
        }

        job, err := client.Jobs.Submit(r.Context(), "Schedule Image capture",
            []workflows.Task{
                &ScheduleImageCapture{
                    Location:      [2]float64{latFloat, lonFloat},
                    ResolutionM:   resolutionM,
                    SpectralBands: spectralBands,
                },
            },
        )
        if err != nil {
            http.Error(w, err.Error(), http.StatusInternalServerError)
            return
        }

        _, _ = io.WriteString(w, fmt.Sprintf("Job submitted: %s\n", job.ID))
    }
}

Copy

In the same way that you can submit jobs across languages you can also submit subtasks across languages.

Creating a Python runner

Write a Python script that starts a runner and registers the ScheduleImageCapture task.

Python

from tilebox.workflows import Client

def main():
    client = Client()
    runner = client.runner(tasks=[ScheduleImageCapture])
    runner.run_forever()

if __name__ == "__main__":
    main()

Copy

Testing it

Start the Go server.

Shell

go run .

Copy

In another terminal, start the Python runner.

Shell

uv run runner.py

Copy

Submit a job to the Go server.

Shell

curl http://localhost:8080/submit?lat=40.75&lon=-73.98&resolution=30&bands[]=489.0,560.6,666.5

Copy

Check the Python runner logs for the following entry:

Logs

Image captured location=[40.75, -73.98] resolution_m=30 spectral_bands=[489, 560.6, 666.5]

Copy

Next Steps

Open in GitHub

The code for this guide is available on GitHub.

As a learning exercise, you can try to change the News API Workflow to replace the FetchNews task with a Go task and keep all the other tasks in Python. You’ll learn how to submit a subtask in another language than what the current task is executed in.