Treasure data digdag visualization
Treasure data is a leading CDP platform and leverages dig-dag for data pipelines within itself. digdag is similar to airflow where we define tasks inside a dag
and is available as an open source. The CDP is a powerful platform but unfortunately no easy way to visualize the pipelines which makes it hard for onboarding
when there are some complex pipelines already built. So I had the below sequence of steps to solve this problem for myself.
- Create Visualization
- Make it colorful and interactive
- Use Graphiviz to create dot diagram
- Render svg and the cmapx
- Create html file and embed svg under `img` tag
- Insert the cmapx into the html after the img tag
- Have the ability to navigate across workflows.
- Update the graph everytime there is a code change.
- Make it part of CICD
pip install -r requirements.txt
The workflows are below and the graphs are interactive:
The scheduled workflows are
- Scheduled Workflows
- Add the script to the root folder under which all digdag projects are listed.
- Execute the python program to generate the graphs.
- Build index.html similar to this.
- Host as a static website.
- Create github workflow to generate graphs when there are changes.
- Identify the changes at the folder level
- Execute the python script to build graph for each folder with a change
- Commit the graphs back to the branch