Skip to content

model_runner

ModelRunner 

ModelRunner(predict_fn, batch_size=1, is_legacy_fn=False)

This class abstracts all the potentially different predict method signatures into a single API that can be used by the model servers.

When initializing the model, pass in a Python function that adheres to any of the defined signatures indicated by chassis.ftypes.PredictFunction type alias. If your model supports batch predictions, set the batch_size to the number of inputs that your model can process at once.

Parameters:

Name Type Description Default
predict_fn PredictFunction

Single predict function of type PredictFunction that represents a model inference function.

 required
batch_size int

Integer representing the batch size your model supports. If your model does not support batching, the default value is 1

 1
is_legacy_fn bool

If True, predict_fn follows legacy format (not typed, only single input and output supported, returns dictionary)

 False

predict_fn instance-attribute 

predict_fn = predict_fn

supports_batch instance-attribute 

supports_batch = batch_size > 1

batch_size instance-attribute 

batch_size = batch_size

legacy instance-attribute 

legacy = is_legacy_fn

load classmethod 

load()

Convenience function used by model servers to load a cloudpickle'd model in the model container.

predict 

predict(inputs)

Performs an inference against the model.

Parameters:

Name Type Description Default
inputs Sequence[Mapping[str, bytes]]

Mapping of input name (str) to input data (bytes) which the predict function is expected to process for inference.

 required

Returns:

Type Description
Sequence[Mapping[str, bytes]]

List of outputs the predict_fn returns

batch 

batch(items, size)

Yields lists of size size until all items have been exhausted.

Parameters:

Name Type Description Default
items Sequence

The batch of inputs to perform inference on.

 required
size int

The batch size that the model supports.

 required