You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
This repository was archived by the owner on Nov 20, 2025. It is now read-only.
Serve machine learning models within a Docker container using Amazon SageMaker.
11
+
## :books: Background
22
12
23
-
-----------------
24
-
Table of Contents
25
-
-----------------
26
-
.. contents::
27
-
:local:
28
-
29
-
Background
30
-
----------
31
-
32
-
`Amazon SageMaker <https://aws.amazon.com/sagemaker/>`__ is a fully managed service for data science and machine learning (ML) workflows.
13
+
[Amazon SageMaker](https://aws.amazon.com/sagemaker/) is a fully managed service for data science and machine learning (ML) workflows.
33
14
You can use Amazon SageMaker to simplify the process of building, training, and deploying ML models.
34
15
35
-
Once you have a trained model, you can include it in a `Docker container<https://www.docker.com/resources/what-container>`__ that runs your inference code.
16
+
Once you have a trained model, you can include it in a [Docker container](https://www.docker.com/resources/what-container) that runs your inference code.
36
17
A container provides an effectively isolated environment, ensuring a consistent runtime regardless of where the container is deployed.
37
-
Containerizing your model and code enables fast and reliable deployment of your model.
18
+
Containerizing your model and code enables fast and reliable deployment of your model.
38
19
39
-
The **SageMaker Inference Toolkit** implements a model serving stack and can be easily added to any Docker container, making it `deployable to SageMaker<https://aws.amazon.com/sagemaker/deploy/>`__.
40
-
This library's serving stack is built on `Multi Model Server<https://github.com/awslabs/mxnet-model-server>`__, and it can serve your own models or those you trained on SageMaker using `machine learning frameworks with native SageMaker support<https://docs.aws.amazon.com/sagemaker/latest/dg/frameworks.html>`__.
41
-
If you use a `prebuilt SageMaker Docker image for inference<https://docs.aws.amazon.com/sagemaker/latest/dg/pre-built-containers-frameworks-deep-learning.html>`__, this library may already be included.
20
+
The **SageMaker Inference Toolkit** implements a model serving stack and can be easily added to any Docker container, making it [deployable to SageMaker](https://aws.amazon.com/sagemaker/deploy/).
21
+
This library's serving stack is built on [Multi Model Server](https://github.com/awslabs/mxnet-model-server), and it can serve your own models or those you trained on SageMaker using [machine learning frameworks with native SageMaker support](https://docs.aws.amazon.com/sagemaker/latest/dg/frameworks.html).
22
+
If you use a [prebuilt SageMaker Docker image for inference](https://docs.aws.amazon.com/sagemaker/latest/dg/pre-built-containers-frameworks-deep-learning.html), this library may already be included.
42
23
43
-
For more information, see the Amazon SageMaker Developer Guide sections on `building your own container with Multi Model Server<https://docs.aws.amazon.com/sagemaker/latest/dg/build-multi-model-build-container.html>`__ and `using your own models<https://docs.aws.amazon.com/sagemaker/latest/dg/your-algorithms.html>`__.
24
+
For more information, see the Amazon SageMaker Developer Guide sections on [building your own container with Multi Model Server](https://docs.aws.amazon.com/sagemaker/latest/dg/build-multi-model-build-container.html) and [using your own models](https://docs.aws.amazon.com/sagemaker/latest/dg/your-algorithms.html).
44
25
45
-
Installation
46
-
------------
26
+
## :hammer_and_wrench: Installation
47
27
48
-
To install this library in your Docker image, add the following line to your `Dockerfile<https://docs.docker.com/engine/reference/builder/>`__:
28
+
To install this library in your Docker image, add the following line to your [Dockerfile](https://docs.docker.com/engine/reference/builder/):
49
29
50
-
.. code:: dockerfile
30
+
```dockerfile
31
+
RUN pip3 install multi-model-server sagemaker-inference-toolkit
32
+
```
51
33
52
-
RUN pip3 install multi-model-server sagemaker-inference-toolkit
34
+
[Here is an example](https://github.com/awslabs/amazon-sagemaker-examples/blob/master/advanced_functionality/multi_model_bring_your_own/container/Dockerfile) of a Dockerfile that installs SageMaker Inference Toolkit.
53
35
54
-
`Here is an example <https://github.com/awslabs/amazon-sagemaker-examples/blob/master/advanced_functionality/multi_model_bring_your_own/container/Dockerfile>`__ of a Dockerfile that installs SageMaker Inference Toolkit.
36
+
## :computer: Usage
55
37
56
-
Usage
57
-
-----
38
+
### Implementation Steps
58
39
59
40
To use the SageMaker Inference Toolkit, you need to do the following:
60
41
61
-
1. Implement an inference handler, which is responsible for loading the model and providing input, predict, and output functions.
62
-
(`Here is an example <https://github.com/aws/sagemaker-pytorch-serving-container/blob/master/src/sagemaker_pytorch_serving_container/default_inference_handler.py>`__ of an inference handler.)
63
-
64
-
.. code:: python
42
+
1. Implement an inference handler, which is responsible for loading the model and providing input, predict, and output functions.
43
+
([Here is an example](https://github.com/aws/sagemaker-pytorch-serving-container/blob/master/src/sagemaker_pytorch_serving_container/default_inference_handler.py) of an inference handler.)
65
44
45
+
```python
66
46
from sagemaker_inference import content_types, decoder, default_inference_handler, encoder, errors
@@ -114,13 +94,13 @@ To use the SageMaker Inference Toolkit, you need to do the following:
114
94
Returns: output data serialized
115
95
"""
116
96
return encoder.encode(prediction, accept)
97
+
```
117
98
118
-
2. Implement a handler service that is executed by the model server.
119
-
(`Here is an example <https://github.com/aws/sagemaker-pytorch-serving-container/blob/master/src/sagemaker_pytorch_serving_container/handler_service.py>`__ of a handler service.)
120
-
For more information on how to define your ``HANDLER_SERVICE`` file, see `the MMS custom service documentation <https://github.com/awslabs/mxnet-model-server/blob/master/docs/custom_service.md>`__.
121
-
122
-
.. code:: python
99
+
2. Implement a handler service that is executed by the model server.
100
+
([Here is an example](https://github.com/aws/sagemaker-pytorch-serving-container/blob/master/src/sagemaker_pytorch_serving_container/handler_service.py) of a handler service.)
101
+
For more information on how to define your `HANDLER_SERVICE`file, see [the MMS custom service documentation](https://github.com/awslabs/mxnet-model-server/blob/master/docs/custom_service.md).
123
102
103
+
``` python
124
104
from sagemaker_inference.default_handler_service import DefaultHandlerService
125
105
from sagemaker_inference.transformer import Transformer
126
106
from sagemaker_pytorch_serving_container.default_inference_handler import DefaultPytorchInferenceHandler
@@ -137,31 +117,34 @@ To use the SageMaker Inference Toolkit, you need to do the following:
3. Implement a serving entrypoint, which starts the model server.
142
-
(`Here is an example <https://github.com/aws/sagemaker-pytorch-serving-container/blob/master/src/sagemaker_pytorch_serving_container/serving.py>`__ of a serving entrypoint.)
143
-
144
-
.. code:: python
122
+
3. Implement a serving entrypoint, which starts the model server.
123
+
([Here is an example](https://github.com/aws/sagemaker-pytorch-serving-container/blob/master/src/sagemaker_pytorch_serving_container/serving.py) of a serving entrypoint.)
`Here is a complete example<https://github.com/awslabs/amazon-sagemaker-examples/tree/master/advanced_functionality/multi_model_bring_your_own>`__ demonstrating usage of the SageMaker Inference Toolkit in your own container for deployment to a multi-model endpoint.
139
+
[Here is a complete example](https://github.com/awslabs/amazon-sagemaker-examples/tree/master/advanced_functionality/multi_model_bring_your_own) demonstrating usage of the SageMaker Inference Toolkit in your own container for deployment to a multi-model endpoint.
157
140
158
-
License
159
-
-------
141
+
## :scroll: License
160
142
161
-
This library is licensed under the `Apache 2.0 License<http://aws.amazon.com/apache2.0/>`__.
162
-
For more details, please take a look at the `LICENSE<https://github.com/aws-samples/sagemaker-inference-toolkit/blob/master/LICENSE>`__ file.
143
+
This library is licensed under the [Apache 2.0 License](http://aws.amazon.com/apache2.0/).
144
+
For more details, please take a look at the [LICENSE](https://github.com/aws-samples/sagemaker-inference-toolkit/blob/master/LICENSE)file.
163
145
164
-
Contributing
165
-
------------
146
+
## :handshake: Contributing
166
147
167
-
Contributions are welcome! Please read our `contributing guidelines <https://github.com/aws/sagemaker-inference-toolkit/blob/master/CONTRIBUTING.md>`__ if you'd like to open an issue or submit a pull request.
0 commit comments