Making ML Deployment Best Practices the Default

By 2022, I’d spent two years shipping NLP models at ScoutinScience¹. I’d learnt that a working model was only part of the job. Making it useful meant adding input validation, model versioning, prediction logs, and an API that could evolve with the application. Each piece was manageable, but wiring them together took some effort. Without a shared approach, our prediction services ended up handling the same concerns in slightly different ways.

For my MSc thesis, I explored how much of this boilerplate could become the default in a shared framework. That’s why I built GreatAI²: a Python framework for taking an inference function from a prototype to a production-ready service.

The API

The starting point is an ordinary Python function. I’ll use a greeting to show the interface without needing to train a model first. Install the package:

pip install great-ai

Then save this as demo.py:

from great_ai import GreatAI

@GreatAI.create
def greeter(name: str) -> str:
    return f"Hello {name}!"

Running great-ai demo.py starts the service on port 6060:

The function is exposed through POST /predict, with its arguments supplied as a JSON object:

curl http://localhost:6060/predict \
  -H 'Content-Type: application/json' \
  -d '{"name": "Andras"}'

The response contains "output": "Hello Andras!" alongside a trace ID, the logged inputs, execution time, and the model versions used. GreatAI also saves this trace, so callers can look up the prediction later by its ID. The dashboard at /dashboard shows the saved records:

I used FastAPI³ for the HTTP layer. GreatAI derives a request schema from the function’s signature and type annotations, using it to validate requests and describe the endpoint in OpenAPI. The generated documentation at /docs lets you inspect and try the API.

The Python interface⁴ remains callable, but its return type changes from str to Trace[str] with the original result being available at .output:

from demo import greeter

trace = greeter("Andras")
assert trace.output == "Hello Andras!"

Both synchronous and asynchronous functions are supported. For offline work, greeter.process_batch(["Andras", "Alice"]) runs a batch in parallel and returns a list of traces. The same inference code can handle individual requests and process a dataset.

Loading a model and validating inputs

For an ML service, the next step is to connect a saved model. Suppose a training script has saved a sentiment classifier with save_model(trained_model, "sentiment"). The first version is numbered 0, which the service can load explicitly:

from great_ai import GreatAI, parameter, use_model

@GreatAI.create
@parameter("text", validate=lambda value: bool(value.strip()))
@use_model("sentiment", version=0)
def predict_sentiment(text: str, model) -> str:
    return str(model.predict([text])[0])

Here, @use_model loads the classifier and supplies the model argument. The HTTP caller only needs to send text and then the model is dependency injected into the service. The decorator also records the loaded version in the trace, linking each prediction to the model used for inference.

The @parameter decorator adds a rule that the type annotation alone cannot express: the text must contain at least one non-whitespace character. The service creation guide⁵ covers these options in detail. The Naive Bayes deployment notebook⁶ brings them together with prediction explanations and batch evaluation.

From feedback to training data

If the sentiment classifier gets a sentence wrong, the application can use the prediction’s trace ID to attach a corrected label through the feedback endpoints. This preserves the original output alongside the correction, so you can compare the service’s answer with the expected label. GreatAI also stores labelled training examples as traces. Its ground-truth API⁷ retrieves those examples alongside predictions that have received feedback. Training code can read both through the same interface, closing the loop from predictions to feedback to new training data.

The storage defaults⁸ let you run this workflow locally, with model files and a trace database on disk. For deployment, you can move models to S3-compatible storage or MongoDB GridFS, and traces to MongoDB. Those storage choices can wait until you’re ready to deploy.

What I learnt

Alongside the implementation, I assembled a catalogue of 33 deployment practices and evaluated the interface with ten practitioners. They deployed a prepared financial-sentiment example and gave positive ratings for ease of use and usefulness. That was encouraging, though a small study of one task says little about long-term adoption. The thesis⁹ covers the method, results, and framework design in detail.

Building GreatAI taught me that defaults are a large part of the API. In practice, they matter far more than the full range of configurations you can express with that API. If a must-do best practice depends on calling a configuration method most users won’t discover, it probably won’t happen. In the end, there’s only so much a pretty website full of best practices can do. But building those practices into sensible defaults actually gets them used. Even if most users never notice the favour, the impact is substantial.

ScoutinScience: https://www.scoutinscience.com/↩
GreatAI: https://great-ai.schmelczer.dev/↩
FastAPI: https://fastapi.tiangolo.com/↩
Python interface: https://great-ai.schmelczer.dev/how-to-guides/use-service/↩
service creation guide: https://great-ai.schmelczer.dev/how-to-guides/create-service/↩
Naive Bayes deployment notebook: https://great-ai.schmelczer.dev/examples/simple/deploy/↩
ground-truth API: https://great-ai.schmelczer.dev/how-to-guides/handle-training-data/↩
storage defaults: https://great-ai.schmelczer.dev/how-to-guides/configure-service/↩
thesis: https://schmelczer.dev/media/downloads/great-ai-andras-schmelczer.pdf ↩