Shared non-Python (e.g. SQL) files?

[elisevansbbfc]

I’m looking at moving 2 separate repos into a monorepo with Polylith - let's call them repo_1 (a dbt project) and repo_2 (a FastAPI app).

Both have shared dependencies, but not only Python packages, there are also SQL files. They live in a folder called my_macros/ in both, e.g.:

my_macros/
├── macro_1.sql
├── macro_2.sql
...

I’d love to put these shared SQL files somewhere inside the Polylith workspace (e.g. under a base like bases/my_namespace/my_macros/) so that both projects can import or open them directly, e.g.:

with open("my_macros/macro_1.sql", "r", encoding="utf-8") as f:
    sql = f.read()

I realise this repo is specific to Python, so sorry if this is like me asking it to do something it's not designed for! But are these kinds of shared non-Python dependencies (like .sql files) supported in a Polylith workspace? If not, is there a recommended workaround or best practice for handling shared SQL resources between projects?

Thanks so much 👍

[DavidVujic]

Since files are not handled the same way as python modules and packages (via the python path), you probably need to figure out the full path of the file you want to open.

How about this?

A component, containing a python module and the sql files. The other components can access the contents of the sql file via the python module:

# this is the "sql" component.
import pathlib

def open_sql_file(name: str) -> str:
    current_folder = pathlib.Path(__file__).parent

    # assuming the sql files are in the same folder in this example
    with open(f"{current_folder}/{name}") as f:
        return f.read()

Any other component or base could import the component and

# in this example, the component with the function has the name "sql"

from my_top_namespace import sql

query = sql.open_sql_file("macro_1.sql")

[elisevansbbfc]

That works really well, thank you!

Just a quick follow-up, is there a similar work-around for integrating with a tool like DBT? With DBT, in dbt_project.yml, I would typically define something like this:

macro-paths: ["macros"]

Which points towards a load of .sql files. However, is it possible to make DBT search for macros in a clean way if the SQL files reside inside a Polylith component (for example, components/my_component/)?

Appreciate that these are all very SQL related questions and might not be what Polylith is intended to do. Thanks again

[DavidVujic]

I have too little knowledge about DBT unfortunately. I looked quickly in the docs: https://docs.getdbt.com/reference/project-configs/macro-paths#default

Paths specified in macro-paths must be relative to the location of your dbt_project.yml file.

Not sure if that means "relative path below the project" or if it allows a relative path sibling or parent (such as ../../)?

[elisevansbbfc]

It does allow ../ e.g. if my dbt_project.yml was in projects/dbt then I could do

macros: "../../components/my_namespace/my_macros"

or even

macros: ["../../components/my_namespace/my_macros", "my_other_macros"]

for multiple paths. This is how I currently use DBT + Polylith for local development, but still unclear on what's best when I use the projects/dbt/Dockerfile for deployment, since I cannot copy or add files in the relative position ../../components/ to the Dockerfile

[DavidVujic]

Aha, I see. Great that it works with relative links, but I understand that it will be a problem in Docker. This is what the poly tool (in combination with uv and hatch) does: copy/transforming the relative paths to in-project paths.

Is your project a DBT project, or is it mainly Python with some DBT macros? If it's mainly a Python project, I would recommend to have the DBT files in a separate structure that works for deployments and Docker.

[DavidVujic]

I don't know if this helps, but I did something similar with Kubernetes configurations (using Kustomize). A Polylith-inspired setup for K8s: https://davidvujic.blogspot.com/2024/08/simple-kubernetes-in-python-monorepos.html