The Data Guy

Airflow PostgreSQL to BigQuery Dynamically Generate DAGs and Tasks


Airflow is now a handy tool for almost all data engineers. If you have a common strategy for the data sync for many data sources, then we can just pass the sources via a loop and it’ll reuse the whole pipeline for everything. Most of you might already know about how to generate the Tasks dynamically. But in this post, Im going to share, how do we dynamically generate the DAGs as well as Tasks in Airflow alo it’ll generate the dag for different sync interval.

JSON config file:

Im not sure how many of you a YAML fans, but I wanted to try with JSON. For YAML also, its a similar logic, but just use the YMAL library to extract the data from the config.


I have 10+ PostgreSQL database servers and I need to sync some of its tables to GCP BigQuery. Every time it has to be an incremental load. On the BigQuery side, the tables are super large (7TB+), so merging the delta data won’t be the right choice. So I did append-only load to the target table. A view is on top of it to de-dup the data. Duplicates on the Main table will be deleted weekly. I Will talk about it in my next blog.


Im following some strict naming conversion for everything.

I have 10+ sources right? So I have the connections on the Airflow with the following conversion.


We can have the naming conversion till IP address, but I had some issues while generating the dynamic tasks, so multiple tasks will get the same task name.

Extracting is not a straightforward one, I had some custom logic to extract the data. So every time, I need to run a specific query for each table and that result will go to BQ. So I created SQL files for every table with the proper SQL select query. And that naming conversion as follows,


These naming conversions will help to dynamically select the SQL files. You’ll get to know once you see the config file.

Finally, the JSON schema file - BQ load from the API must need a JSON schema file to load the data into the table, So I have generated such schema files with the following naming conversion.


Meta Table:

In my case, I want to maintain everything on the GCP side, the source databases were on the on-prem side. And I don’t want to spin up a VM or CloudSQL just to store the metadata. So I used the BQ table to store the metadata. In the Metadata table, I keep track of the max_timestamp value of the last load. So next time, the highest value will be pick from that.

Note: Make sure the value for the datasource_dbconn must be your Airflow connection id. Because the naming conversion will play a major role here.

Here is the table DDL.

-- databaseid and datasource_dbconn is used in our use case
-- you can skip those columns
  bqdataset.tablemeta (id INT64 NOT NULL,
    tablename STRING NOT NULL,
    datbabaseid INT64,
    datasource_dbconn STRING NOT NULL,
    maxvalue STRING NOT NULL)

Custom PostgreSQL Operator:

The native PostgreSQL to GCS operator is not converting the timestamp and some other columns properly. So I wanted to make some changes to this operator and ended up creating a custom operator called custom_PostgresToGCSOperator. You can read more about this and get that operator here

SQL file to extract max timestamp value from source:

We generally don’t use select max(ts) from table because we want to sync some specific records only. And for each table the logic is different. So in this DAG, I used SQL files for each table to find the max timestamp value. Its again optional, but replace the code accordingly.

Naming Conversion:

-- Sample SQL to find the max ts 
SELECT cast(Max({}) as varchar(40))
FROM   trans.movie_subscriptions 
WHERE  database_id=1 and state='success'
and {}>='{}';

These to {} will be the max ts value column name and this value from the meta table.

SQL file for export:

Data export also not the simple select *. I had to export via some specific SQL queries. So all the export processes will read the SQL file and export the data in CSV into GCS.

Naming Conversion:

-- sample export SQL
		when tbl1.jcol::json ->>'renewal' = '' then 0
		else (tbl1.jcol::json ->>'renewal')::int
	end renewal,
	transactions.pat tbl1
inner join transactions.job tbl2 on = tbl2.params_id
	tbl1.update_timestamp >= '{}' and tbl1.update_timestamp <='{}';

Config File Template:

I have the data sync for different tables and different time interval. So this template should have the following information.

My config file template:

	"2mins": {
		"platform.vouchers.movie_vouchers": ["updated_at", "prod_postgres_10.10.10.71_platform"],
		"databaseid_1.plan.subscription": ["update_timestamp", "prod_postgres_10.10.10.82_source"],
		"databaseid_2.plan.subscription": ["update_timestamp", "prod_postgres_10.10.10.86_subscriptions"]
	"30mins": {
		"": ["update_timestamp", "prod_postgres_10.10.10.73_micorservices"],
		"": ["update_timestamp", "prod_postgres_10.10.10.73_micorservices"],
		"databaseid_1.transactions.payment": ["update_timestamp", "prod_postgres_10.10.10.82_source"],
		"databaseid_2.transactions.payment": ["update_timestamp", "prod_postgres_10.10.10.191_source"],
		"databaseid_1.transactions.gateway": ["update_timestamp", "prod_postgres_10.10.10.82_source"],
		"databaseid_2.transactions.gateway": ["update_timestamp", "prod_postgres_10.10.10.191_source"],
		"databaseid_1.transactions.master": ["update_timestamp", "prod_postgres_10.10.10.82_source"],
		"databaseid_2.transactions.master": ["update_timestamp", "prod_postgres_10.10.10.191_source"]

Here, I have added some prefixes (platform.vouchers.movie_vouchers) before the table and schema name.

Im going to generate the tasks based on this JSON KEY. So if we have the same schema and table name for multiple databases, the last task will replace the previous one. So the task must be unique, that’s why I added this prefix for some reference.

Pipeline Flow:

Airflow pipeline

DAG Code:

Things to replace:

You can find the complete code base here, if you have any clear details, find it in the example codebase.

· gcp, bigquery, postgresql


Loading Comments