`Treasure data digdag visualization`

Treasure data is a leading CDP platform and leverages dig-dag for data pipelines within itself. digdag is similar to airflow where we define tasks inside a dag and is available as an open source. The CDP is a powerful platform but unfortunately no easy way to visualize the pipelines which makes it hard for onboarding when there are some complex pipelines already built. So I had the below sequence of steps to solve this problem for myself.

Solution Approach

Create Visualization
Make it colorful and interactive

Use Graphiviz to create dot diagram
Render svg and the cmapx
Create html file and embed svg under `img` tag
Insert the cmapx into the html after the img tag

Have the ability to navigate across workflows.
Update the graph everytime there is a code change.
Make it part of CICD

Installation

pip install -r requirements.txt

The workflows are below and the graphs are interactive:

The scheduled workflows are

Scheduled Workflows

Usage example

Add the script to the root folder under which all digdag projects are listed.
Execute the python program to generate the graphs.
Build index.html similar to this.
Host as a static website.

TODO

Create github workflow to generate graphs when there are changes.
Identify the changes at the folder level
Execute the python script to build graph for each folder with a change
Commit the graphs back to the branch

Github Link

All source code here

References

https://github.com/y-abe/digdag-graph