chore(alertflow): bumping airflow version and implementing prod changes #28
Conversation
| schedule='@daily', | ||
| default_args=DEFAULT_ARGS, | ||
| start_date=pendulum.datetime(2014, 1, 1), | ||
| start_date=pendulum.datetime(2023, 8, 1), |
There was a problem hiding this comment.
If the start date is expected to change over time, perhaps it would be better to create a function that calculates the initial date dynamically, instead of hardcoding it in the code.
There was a problem hiding this comment.
This start date is only important to reduce the amount of days the DAG will have to check for incomplete data. I've ran many times locally and I'm sure there's no missing data before 2023/aug. but for now on, it thing it would be good to keep this date, then every time the container is rebuilt, it will check for possible missing data
.github/workflows/build.yaml
Outdated
| branches: [ main ] | ||
|
|
||
| env: | ||
| BASE_URL_HOST=localhost:8081 |
There was a problem hiding this comment.
Even though this is a CI variable, if you need to access this host from another container, it might be a good idea to use the container's hostname here.
| && /home/airflow/mambaforge/bin/mamba create -y -n episcanner-downloader python=3.11 poetry psycopg2 python-dotenv \ | ||
| && chown -R ${HOST_UID}:${HOST_GID} ${AIRFLOW_HOME}/ /home/airflow/mambaforge/ | ||
|
|
||
| RUN echo "alias activate_episcanner='source /home/airflow/mambaforge/bin/activate episcanner-downloader'" >> /home/airflow/.bashrc |
luabida
force-pushed
the
environment-config
branch
from
4d6375f to
d51c452
Compare
luabida
force-pushed
the
environment-config
branch
from
d51c452 to
55b9d2d
Compare
2 participants
No description provided.