│ │ Commented on Sun Jul 20 08:39:58 2025 UTC
│ │
│ │ I work in the field at which Astronomer products are targeted, so here's a simplistic explanation. Maybe it demystifies some jargon for people.
│ │
│ │ Businesses tend to have a lot of data, and that data is usually in different databases. This is for many reasons - sometimes one database is well suited for being the backbone of a website but not suited for complex statistics, so they get another one. Sometimes they have a messy corporate structure where different teams build independently and use different databases independently of each other. Anyway, they usually want to move data between these databases. Moving data is known as "ETL", and the application that does it is known as "ETL pipeline". (There's also "ELT", but we'll skip that.)
│ │
│ │ Companies tend to want to have complex inter-dependent logic in these ETL pipelines. Say, you launch the process on Sundays only at 1:00 AM (regardless of daylight savings), move data from database A to database B, but if it contains a certain value you also want to take more data from databases C and D, and then you want to produce a PDF report and email it to person E. The application that handles this logic and triggers different parts of it is known as "pipeline orchestrator".
│ │
│ │ The most popular orchestrator product is Apache Airflow. Airbnb built it for their own needs in mid-10s, and then published all the code for free. It is still actively developed by volunteer maintainers under Apache Foundation. So it is open source and anyone can run it. You can run it on your laptop right now if you wanted to. Airflow is not the only product, and there are alternatives both older and younger (Control-M, Kestra, Prefect, Dagster, Apache Oozie, ...), but still the most popular.
│ │
│ │ Airflow has some quirks. For one, it has notoriously shyt documentation. For two, it is kind of a pain to deploy it (as in, upload new code for more pipelines on an existing and running installation). It is also not very stable at scale (when you run a lot of pipelines it can crash). A company that runs Airflow can tweak their installation and build tools to solve these problems, and many do. But if they don't want to spend time on it, Astronomer - the company - builds their own "modified" version of Airflow that solves these problems. That version is not free, and a company will pay Astronomer for a license to run it. IIRC you can also pay Astronomer even more and have them run it fully for you.
│ │
│ │ They are not the only ones who do it, AWS and GCP have their own "modified easier to use Airflow" products (MWAA and Cloud Composer respectively). Some Astronomer employees actually are maintainers of open-source Airflow and the company still contributes features to the open-source codebase too. They also write their own alternative documentation, and it is free and public (I work with open-source Airflow and read Astronomer docs all the time instead of official ones).
│ │
