Skip to content

Conversation

@hrodmn
Copy link
Contributor

@hrodmn hrodmn commented Jan 10, 2025

After #72 the Docker build created a Lambda asset that was greater than 262 MB unzipped 🙀

In this PR I am trying a few things to fix it:

  • switch to uv for dependency management
  • move requirements-cdk.txt into the deployment dependency group
  • clean up the dependencies in pyproject.toml, making sure ones we only need for tests are not listed in the default dependencies (I think this actually fixed it)
  • run cdk synth in CI and check the size of the assets in cdk.out

With these changes we squeak under the package size limit by 2 MB 😬

I compared the asset package size to titiler-cmr (which has a 230 MB asset size) and the main difference in titiler-multidim is the addition of zarr as a dependency which adds numcodecs which demands 33MB in the asset directory. This package is full of hefty dependencies!

Resolves #56 (for now)

Here is a summary of the largest dependencies and the size they occupy in the cdk Lambda asset:

henry@quercus:~/workspace/devseed/titiler-xarray-application/infrastructure/aws/cdk.out/asset.bcc78660745bbaa8bbfcec1da064c2564455348ac653ec35f5bf2eb9ebdbb979$ du -h --max-depth=1 | sort -hr
261M    .
47M     ./rasterio.libs
33M     ./numcodecs
30M     ./pandas
25M     ./numpy.libs
24M     ./numpy
18M     ./pyproj
16M     ./rasterio
13M     ./pyproj.libs
9.3M    ./h5py
5.6M    ./aiohttp
5.1M    ./h5py.libs
4.6M    ./pydantic_core
4.5M    ./cftime
3.1M    ./xarray
2.7M    ./tzdata
2.7M    ./pytz
1.5M    ./pydantic
1.3M    ./numexpr
1.1M    ./rio_tiler
1.1M    ./redis
...

@hrodmn hrodmn requested a review from vincentsarago January 13, 2025 18:04
@hrodmn
Copy link
Contributor Author

hrodmn commented Jan 13, 2025

With the C/C++ strip operation in the Lambda Dockerfile I have the /asset size down to ~215 MB which gives us plenty of headroom. If anyone would prefer I add that change in a separate PR I think we could get away with the pyproject.toml cleanup alone, but we would need to add more specific version constraints to avoid package updates causing the asset size to increase!

@hrodmn hrodmn merged commit 0f31041 into dev Jan 13, 2025
5 checks passed
@hrodmn hrodmn deleted the fix/lambda-docker-size branch January 13, 2025 19:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Package size exceeding lambda limits

3 participants