3.1

Package a Docker Image for Model Deployment (Python)

Enterprise Applicable to Enterprise tier only

This doc shows how to package a model into a format-valid docker image for the PrimeHub model deployment feature.

The PrimeHub model deployment feature is based on Seldon. This doc takes reference from Seldon official documentations and other resources which are listed in the last part.

Prerequisites

Check everything is ready to go by running:

s2i usage seldonio/seldon-core-s2i-python3:0.18

Prepare the Model and Code (Python)

  • Please use Python 3.6 (Recommended)

  • Create a requirements.txt file and write down all required packages

    keras
tensorflow
numpy
...

  • Create a .s2i folder and create a .s2i/environment file with the following content:

    MODEL_NAME=MyModel
API_TYPE=REST
SERVICE_TYPE=MODEL
PERSISTENCE=0

  • Create a MyModel.py file with the following example template:

    class MyModel(object):
    """
    Model template. 
    You can load your model parameters in __init__ from a location accessible at runtime.
    """

    def __init__(self):
        """
        Add any initialization parameters. These will be passed at runtime from the graph definition parameters 
        defined in your seldondeployment kubernetes resource manifest.
        """
        print("Initializing")

    def predict(self, X, features_names=None):
        """
        Return a prediction.

        Parameters
        ----------
        X : array-like
        feature_names : array of feature names (optional)
        """
        print("Predict called - will run identity function")
        return X
    • File and class name MyModel should be the same as MODEL_NAME under .s2i/environment
    • Load or initiate your model under the __init__ function
    • The predict method takes a numpy-array X and list of string feature_names (optional), then returns an array of predictions (the return array should be at least 2-dimensional)

    More detailed information on how to write the Python file for model deployment in different frameworks, please refer to the section Example Codes for Different Frameworks.

Build the Image

Make sure you are in the folder that includes requirements.txt, .s2i/environment, python file for model deployment, and model file...etc.

If this folder is managed by Git, please commit all changes into the git.

We will use seldonio/seldon-core-s2i-python3:0.18 as a base image, installing environment and packaging our model file into the target, my-model-image. You can use the following command to package the docker image:

(Using seldonio/seldon-core-s2i-python3 instead if using Python 3 rather than Python 3.6)

 s2i build . seldonio/seldon-core-s2i-python3:0.18 my-model-image

Then check the image by docker images.

REPOSITORY                         TAG                 IMAGE ID            CREATED             SIZE
my-model-image                     latest              4a0f28ee4f4c        3 minutes ago       1.66GB
seldonio/seldon-core-s2i-python3   0.18                0380e4efa66e        7 weeks ago         794MB

Test the Image

In order to make sure your model image is well packaged, you can run your model as a Docker container locally:

docker run -p 5000:5000 --rm my-model-image

and curl:

    curl -X POST localhost:5000/api/v1.0/predictions \
         -H 'Content-Type: application/json' \
         -d '{ "data": { "ndarray": [[5.964,4.006,2.081,1.031]]}}'

Replace ndarray content in curl example according to your application.

You have built the docker image for a PrimeHub model deployment successfully now.

Push the Image

Next, push the image into the docker hub (or other docker registries) and check PrimeHub tutorial to serve the model under PrimeHub.

Tag your docker image:

docker tag my-model-image test-repo/my-model-image

Then push to docker registry:

docker push test-repo/my-model-image

(Optional) Example Codes for Different Frameworks

Here are some Python snippets of how to export a model file then load it and run the prediction in another file. By following the Python wrapper format, PrimeHub supports various popular ML frameworks to serve models.

Tensorflow 1

  • Output a model file model/deep_mnist_model

        saver = tf.train.Saver()
    saver.save(sess, "model/deep_mnist_model")

  • MyModel.py, load a model and run a prediction

        import tensorflow as tf
    import numpy as np
    import os
    
    class DeepMnist(object):
        def __init__(self):
            self.loaded = False
            
        def load(self):
            print("Loading model",os.getpid())
            self.sess = tf.Session()
            saver = tf.train.import_meta_graph("model/deep_mnist_model.meta")
            saver.restore(self.sess,tf.train.latest_checkpoint("./model/"))
            graph = tf.get_default_graph()
            self.x = graph.get_tensor_by_name("x:0")
            self.y = graph.get_tensor_by_name("y:0")
            self.loaded = True
            print("Loaded model")
            
        def predict(self,X,feature_names):
            if not self.loaded:
                self.load()
            predictions = self.sess.run(self.y,feed_dict={self.x:X})
            return predictions.astype(np.float64)

Tensorflow 2

  • Output a model file 1

        model.save("1")

  • MyModel.py, load a model and run a prediction

        import tensorflow as tf
    
    class MNISTModel:
        def __init__(self):
            self._model = tf.keras.models.load_model('1')
    
        def predict(self, X, feature_names=None, meta=None):
            output = self._model.predict(X)
            probability = output[0]
            predicted_number = tf.math.argmax(probability)
            return {"predicted_number": predicted_number.numpy().tolist(), "probability": probability.tolist()}

Keras

  • Output a model file keras-mnist.h5

        model.save('keras-mnist.h5')

  • MyModel.py, load a model and run a prediction

        from keras.models import load_model
    from PIL import Image
    from io import BytesIO
    import numpy as np
    
    class MyModel(object):
        def __init__(self):
            self.model = load_model('keras-mnist.h5')
            self.model._make_predict_function()
            
        def predict(self,X,features_names):
            imageStream = BytesIO(X)
            image = Image.open(imageStream).resize((28, 28)).convert('L')
            data = np.asarray(image)
            data = np.expand_dims(data, axis=0)
            data = np.expand_dims(data, axis=-1)
            return self.model.predict(data)

Scikit-learn

  • Output a model file IrisClassifier.sav

        joblib.dump(p, "IrisClassifier.sav")

  • MyModel.py, load a model and run a prediction

        from sklearn.externals import joblib
    
    class IrisClassifier(object):
    
        def __init__(self):
            self.model = joblib.load('IrisClassifier.sav')
    
        def predict(self,X,features_names):
            return self.model.predict_proba(X)

Pytorch

  • Output a model file mnist_cnn.pt

        torch.save(model.state_dict(), "mnist_cnn.pt")

  • MyModel.py, load a model and run a prediction

        import torch
    import torch.nn as nn
    import torch.nn.functional as F
    
    class Net(nn.Module):
        def __init__(self):
            super(Net, self).__init__()
            self.conv1 = nn.Conv2d(1, 32, 3, 1)
            self.conv2 = nn.Conv2d(32, 64, 3, 1)
            self.dropout1 = nn.Dropout2d(0.25)
            self.dropout2 = nn.Dropout2d(0.5)
            self.fc1 = nn.Linear(9216, 128)
            self.fc2 = nn.Linear(128, 10)
    
        def forward(self, x):
            x = self.conv1(x)
            x = F.relu(x)
            x = self.conv2(x)
            x = F.relu(x)
            x = F.max_pool2d(x, 2)
            x = self.dropout1(x)
            x = torch.flatten(x, 1)
            x = self.fc1(x)
            x = F.relu(x)
            x = self.dropout2(x)
            x = self.fc2(x)
            output = F.softmax(x, dim=1)
            return output
    
    class MNISTModel:
        def __init__(self):
            self._model = Net()
            self._model.load_state_dict(torch.load("mnist_cnn.pt"))
            self._model.eval()
    
        def predict(self, x, names):
            output = self._model(torch.from_numpy(x).float())
            return {"probability": output.tolist()}

XGBoost

  • Output a model file xgboost.model

        bst = xgb.train(...)
    bst.save_model('xgboost.model')

  • MyModel.py, load a model and run a prediction

        import xgboost as xgb
    
    class MyModel(object):
        def __init__(self):
            self.bst = xgb.Booster({'nthread':4})
            self.bst.load_model("xgboost.model") 
            
        def predict(self,X,features_names):
            dtest = xgb.DMatrix(X)
            return self.bst.predict(dtest)

MXNet

  • Output a model file mx-model___

        model_prefix = 'mx-model'
    checkpoint = mx.callback.do_checkpoint(model_prefix)
    mod.fit(..., epoch_end_callback=checkpoint)

  • MyModel.py, load a model and run a prediction

        import mxnet as mx
    from PIL import Image
    import numpy as np
    from io import BytesIO
    
    class MyModel(object):
        def __init__(self):
            model_prefix = 'mx-model'
            epoch_num = 2
            self.model = mx.mod.Module.load(model_prefix, epoch_num)
            
            data_shape = [("data", (1, 28, 28, 1))]
            label_shape = [("softmax_label", (1,))]
            self.model.bind(data_shape, label_shape)
    
        def predict(self,X,features_names):
            imageStream = BytesIO(X)
            image = Image.open(imageStream).resize((28, 28)).convert('L')
    
            data = np.asarray(image)
            data = np.expand_dims(data, axis=0)
            data = np.expand_dims(data, axis=-1)
            return self.model.predict(data).asnumpy()

LightGBM

  • Output a model file model.pkl

        gbm = lgb.train(...)
    with open('model.pkl', 'wb') as fout:
        pickle.dump(gbm, fout)

  • MyModel.py, load a model and run a prediction

        import pickle
    
    class MyModel(object):
        def __init__(self):
            with open('model.pkl', 'rb') as fin:
                self.pkl_bst = pickle.load(fin)
    
        def predict(self,X,features_names):
            return self.pkl_bst.predict(X)

Reference

← PreviousNext →