Motivation
The idea is to relieve developers of the
burden of managing the cluster and
offer a REST API to interact, similar to tools like Airflow but
without Python-specific constraints.
In the future, genie-do will support
libraries in various languages, enabling custom local actions similar
to GitHub Actions, where the community can build and contribute their
own actions.
This platform aims to bring the functionality of GitHub Actions + AWS
Lambda + Apache Spark together, enabling seamless orchestration and automation.
Genie-do is designed as a distributed
executor in Kubernetes, orchestrated with Apache Airflow, and is
deployable on any cloud provider or on-premises environment. If you
can represent your job as a Docker image or containerized application,
we will run it, retry it, distribute it, and much more.
You’ll also have access to Kafka, S3, Apache Spark & Flink,
PostgreSQL, NoSQL, and other essential technologies, supporting
real-time streaming, data pipelines, and event-driven architecture.
Supporting gRPC services and REST API management, with libraries in
multiple languages, genie-do makes it
easy for you to run and manage your code effortlessly, with built-in
scalability, fault tolerance, and secure integration across distributed
systems.
