[CI] Create nightly images/wheels

[simon-mo]

Anything you want to discuss about vllm.

It would be great if we can distribute nightly images and wheels. This should be as simple as building images to push them to google container registry w/ public access, and push wheels to storage bucket.

[youkaichao]

At least we should do this against nightly build pytorch, every day or every several days. If pytorch has some breaking change, we will know in less than one week.

[simon-mo]

This is a different task. @youkaichao can you create a new issue tracking it?

[youkaichao]

issue created at #3560 .

BTW, to build wheels, we also have to consider our support matrix, across cuda version, nccl version, pytorch version, etc.

[simon-mo]

Our currently support matrix is pretty simple! which is tie to one single version of each

[rkooo567]

+1 on this. It worked very well with ray, especially when users have a bug that is fixed in the master.

[simon-mo]

Here are some pointers, @rkooo567

• Wheels are currently not build in Buildkite, rather it is in GitHub Actions: https://github.com/vllm-project/vllm/blob/main/.github/workflows/publish.yml. We would like the wheels to be built in Buildkite environment.
• In our buildkite CI, the following code builds docker container in a big CPU machine:

  vllm/.buildkite/test-template.j2

  Lines 11 to 20 in dfeb2ec

  	- label: ":docker: build image"
  	commands:
  	- "docker build --build-arg max_jobs=16 --tag {{ docker_image }} --target test --progress plain ."
  	- "docker push {{ docker_image }}"
  	env:
  	DOCKER_BUILDKIT: "1"
  	retry:
  	automatic:
  	- exit_status: -1 # Agent was lost
  	limit: 5

  . And then the tests are kicked off inside docker containers running in K8s

  vllm/.buildkite/test-template.j2

  Lines 23 to 64 in dfeb2ec

  	{% for step in steps %}
  	- label: "{{ step.label }}"
  	agents:
  	queue: kubernetes
  	soft_fail: {{ step.soft_fail or false }}
  	{% if step.parallelism %}
  	parallelism: {{ step.parallelism }}
  	{% endif %}
  	retry:
  	automatic:
  	- exit_status: -1 # Agent was lost
  	limit: 5
  	plugins:
  	- kubernetes:
  	podSpec:
  	volumes:
  	- name: dshm
  	emptyDir:
  	medium: Memory
  	containers:
  	- image: "{{ docker_image }}"
  	command: ["bash"]
  	args:
  	- '-c'
  	- "'cd {{ (step.working_dir or default_working_dir) | safe }} && {{ step.command or (step.commands | join(' && ')) | safe }}'"
  	{% if not step.no_gpu %}
  	resources:
  	requests:
  	nvidia.com/gpu: "{{ step.num_gpus or default_num_gpu }}"
  	limits:
  	nvidia.com/gpu: "{{ step.num_gpus or default_num_gpu }}"
  	{% endif %}
  	env:
  	- name: HF_TOKEN
  	valueFrom:
  	secretKeyRef:
  	name: hf-token-secret
  	key: token
  	volumeMounts:
  	- mountPath: /dev/shm
  	name: dshm
  	{% endfor %}

  The for loop iterate over this yaml lists https://github.com/vllm-project/vllm/blob/main/.buildkite/test-pipeline.yaml
• The CPU machine currently builds the test image

  vllm/Dockerfile

  Lines 79 to 95 in dfeb2ec

  	#################### TEST IMAGE ####################
  	# image to run unit testing suite
  	FROM dev AS test
  	# copy pytorch extensions separately to avoid having to rebuild
  	# when python code changes
  	WORKDIR /vllm-workspace
  	# ADD is used to preserve directory structure
  	ADD . /vllm-workspace/
  	COPY --from=build /workspace/vllm/*.so /vllm-workspace/vllm/
  	# Install flash attention (from pre-built wheel)
  	RUN --mount=type=bind,from=flash-attn-builder,src=/usr/src/flash-attention-v2,target=/usr/src/flash-attention-v2 \
  	pip install /usr/src/flash-attention-v2/*.whl --no-cache-dir
  	# ignore build dependencies installation because we are using pre-complied extensions
  	RUN rm pyproject.toml
  	RUN --mount=type=cache,target=/root/.cache/pip VLLM_USE_PRECOMPILED=1 pip install . --verbose
  	#################### TEST IMAGE ####################

  , not the official image we distribute.

  vllm/Dockerfile

  Lines 120 to 131 in dfeb2ec

  	#################### OPENAI API SERVER ####################
  	# openai api server alternative
  	FROM vllm-base AS vllm-openai
  	# install additional dependencies for openai api server
  	RUN --mount=type=cache,target=/root/.cache/pip \
  	pip install accelerate hf_transfer modelscope
  	COPY --from=build /workspace/vllm/*.so /workspace/vllm/
  	COPY vllm vllm
  	ENTRYPOINT ["python3", "-m", "vllm.entrypoints.openai.api_server"]
  	#################### OPENAI API SERVER ####################

For nightly wheels, we should replicate the environment (maybe use conda?) to perform wheel building process. I haven't figure out whether we should run this on big CPU machine, small CPU containers, or small GPU containers. I would try maybe a small CPU container to build wheels if needed to avoid the big CPU machine from being queued up. In order to use a small CPU container, you can probably just create a jobs similar to other test but use num_gpus=0. The K8s cluster running the tests might need extra setup to push the wheels to a storage bucket.

For nightly container images, we need to use the CPU machine to build the openai target and push to the gcr destination as well. perhaps as a separate repo in the google cloud container registry.

Note that both of these, we don't need to publish them in official container registry (docker.io) nor package registry (pypi) for vLLM.

[rkooo567]

I may have time to look at it after #3631

[github-actions]

This issue has been automatically marked as stale because it has not had any activity within 90 days. It will be automatically closed if no further activity occurs within 30 days. Leave a comment if you feel this issue should remain open. Thank you!

[youkaichao]

close as we have per-commit wheels now.