Home

Awesome

Due to new featues that have since been added to Azure Machine Learning, this repository is now deprecated.

AI for Earth - Creating APIs

These images and examples are meant to illustrate how to build containers for use in the AI for Earth API system. The following images and tags (versions/images) are available on Dockerhub:

Notice

Additional to a running docker environment, GPU images require NVIDIA Docker package to support CUDA.

CUDA Toolkit

To view the license for the CUDA Toolkit included in the cuda base image, click here

CUDA Deep Neural Network library (cuDNN)

To view the license for cuDNN included in the cuda base image, click here

Contents

  1. Repo Layout
  2. Quickstart Tutorial
    1. Choose a base image or example
    2. Insert code to call your model
    3. Input handling
    4. Output handling
    5. Function decorator detail
    6. Create AppInsights instrumentation keys
    7. Install required packages
    8. Set environment variables
    9. Build and run your image
    10. Make requests
    11. Publish to Azure Container Registry
    12. Run your container in ACI
    13. FAQs
  3. Contributing

Repo Layout

Notes

Quickstart Tutorial

This quickstart will walk you through turning a model into an API. Starting with a trained model, we will containerize it, deploy it on Azure, and expose an endpoint to call the API. We will leverage Docker containers, Azure Application Insights, Azure Container Registry, and Azure Container Instances.

We are assuming that you have a trained model that you want to expose as an API. To begin, download or clone this repository to your local machine.

Create an Azure Resource Group

Throughout this quickstart tutorial, we recommend that you put all Azure resources created into a single new Resource Group. This will organize these related resources together and make it easy to remove them as a single group.

From the Azure Portal, click Create a resource from the left menu. Search the Marketplace for "Resource Group", select the resource group option and click Create.

Search for Resource Group

Use a descriptive resource group name, such as "ai4e_yourname_app_rg". Then click Create.

Resource Group

Machine Setup

You will need an active Azure subscription as well as the following software installed.

Choose a base image or example

AI for Earth APIs are all built from an AI for Earth base image. You may use a base image directly or start with an example. The following sections will help you decide.

Base images

Examples

In general, if you're using Python, you will want to use an image or example with the base-py or blob-py images. If you are using R, you will want to use an image or example with the base-r or blob-r images. The difference between them: the blob-* image contains everything that the cooresponding base-* image contains, plus additional support for mounting Azure blob storage. This may be useful if you need to process (for example) a batch of images all at once; you can upload them all to Azure blob storage, the container in which your model is running can mount that storage, and access it like it is local storage.

Asynchronous (async) vs. Synchronous (sync) Endpoint

In addition to your language choice, think about whether your API call should be synchronous or asynchronous.

Asynchronous Implementation Examples

The following examples demonstrate async endpoints:

Synchronous Implementation Examples

The following examples demonstrate sync endpoints:

Input/Output Patterns

While input patterns can be used for sync or async designs, your output design is dependent on your sync/async choice, therefore, we have identified recommended approaches for each.

Input Recommendations

JSON

JSON is the recommended approach for data ingestion.

Binary Input

Many applications of AI apply models to image/binary inputs. Here are some approaches:

Asynchronous Pattern

The preferred way of handling asynchronous API calls is to provide a task status endpoint to your users. When a request is submitted, a new taskId is immediately returned to the caller to track the status of their request as it is processed.

We have several tools to help with task tracking that you can use for local development and testing. These tools create a database within the service instance and are not recommended for production use.

Once a task is completed, the user needs to retrieve the result of their service call. This can be accomplished in several ways:

Examples

We have provided several examples that leverage these base images to make it easier for you to get started.

After you've chosen the example that best fits your scenario, make a copy of that directory, which you can use as your working directory in which you apply your changes.

Insert code to call your model

Next, in your new working directory, we need to update the example that you chose with code to call your specific model. This should be done in the runserver.py file (if you are using a Python example) or the api_example.R file (if you are using an R example) in the my_api (or similarly named) subfolder.

Input handling

Your model has inputs and outputs. For example, let's consider a classification model that takes an image and classifies its contents as one of multiple species of animal. The input that you need to provide to this model is an image, and the output that you provide may be JSON-formatted text of the classifications and their confidence.

Some examples of how to send parameters as inputs into your APIs follow.

GET URL parameters

For GET operations, best practice dictates that a noun is used in the URL in the segment before the related parameter. An echo example is as follows.

Python and Flask
@ai4e_service.api_sync_func(api_path = '/echo/<string:text>', methods = ['GET'], maximum_concurrent_requests = 1000, trace_name = 'get:echo', kwargs = {'text'})
def echo(*args, **kwargs):
    return 'Echo: ' + kwargs['text']
R and Plumber
#* @param text The text to echo back
#* @get /echo/<text>
GetProcessDataTaskStatus<-function(text){
  print(text)
}

POST body

For non-trivial parameters, retrieve parameters from the body sent as part of the request. JSON is the preferred standard for API transmission. The following gives an example of sample input, followed by Python and R usage.

Sample Input
{
    "container_uri": "https://myblobacct.blob.core.windows.net/user?st=2018-08-02T12%3A01%3A00Z&se=5200-08-03T12%3A01%3A00Z&sp=rwl&sv=2017-04-17&sr=c&sig=xxx",
    "run_id": "myrunid"
}
Python and Flask
from flask import Flask, request
import json

post_body = request.get_json()

print(post_body['run_id'])
print(post_body['container_uri'])
R and Plumber
library(jsonlite)

#* @post /process-data
ProcessDataAPI<-function(req, res){
  post_body <- req$postBody
  input_data <- fromJSON(post_body, simplifyDataFrame=TRUE)

  print(input_data$run_id)
  print(input_data$container_uri)
}

Output handling

Then, you need to send back your model's results as output. Two return types are important when dealing with hosted ML APIs: non-binary and binary.

Non-binary data

You may need to return non-binary data, like simple strings or numbers. The preferred method to return non-binary data is to use JSON.

Python and Flask
import json
def post(self):
    ret = {}
    ret['run_id'] = myrunid   
    ret['container_uri'] = 'https://myblobacct.blob.core.windows.net/user?st=2018-08-02T12%3A01%3A00Z&se=5200-08-03T12%3A01%3A00Z&sp=rwl&sv=2017-04-17&sr=c&sig=xxx'

    return json.dumps(ret)
R and Plumber
ProcessDataAPI<-function(req, res){
  post_body <- req$postBody
  input_data <- fromJSON(post_body, simplifyDataFrame=TRUE)

  # Return JSON containing run_id and container_uri
  data.frame(input_data$run_id, input_data$container_uri)
}

Binary data

You may also need to return binary data, like images.

Python and Flask
from io import BytesIO
import tifffile
from flask import send_file

ACCEPTED_CONTENT_TYPES = ['image/tiff', 'application/octet-stream']

if request.headers.get("Content-Type") in ACCEPTED_CONTENT_TYPES:
    tiff_file = tifffile.imread(BytesIO(request.data))
    # Do something with the tiff_file...
    prediction_stream = BytesIO()
    # Create your image to return...
    prediction_stream.seek(0)
    return send_file(prediction_stream)

Function decorator detail

We use function decorators to create APIs out of your functions, such as those that execute a model. Here, we will detail the two decorators and their parameters.

There are two decorators:

Each decorator contains the following parameters:

Create AppInsights instrumentation keys

Application Insights is an Azure service for application performance management. We have integrated with Application Insights to provide advanced monitoring capabilities. You will need to generate both an Instrumentation key and an API key to use in your application.

The instrumentation key is for general logging and tracing. This is found under the "Properties" section for your Application Insights instance in the Azure portal.

Search for App Insights

Click Create, then choose a name for your Application Insight resource. For Application Type, choose General from the drop-down menu. For Resource Group, select "Use existing" and choose the resource group that you created earlier.

Create App Insights

Once your AppInsights resource has successfully deployed, navigate to the resource from your home screen, and locate the Instrumentation Key.

Get Instrumentation Key

Next, create a Live Metrics API key. Scroll down in the left menu to find API Access within Application Insights, and click Create API key. When creating the key, be sure to select "Authenticate SDK control channel".

Get API Key

Get API Key

Copy and store both of these keys in a safe place.

Install required packages

Now, let's look at the Dockerfile in your code. Update the Dockerfile to install any required packages. There are several ways to install packages. We cover popular ones here:

RUN /usr/local/envs/ai4e_py_api/bin/pip install grpcio opencensus

RUN apt-get install gfortran -y

RUN R -e 'install.packages("rgeos"); library(rgeos)'

Set environment variables

The Dockerfile also contains several environment variables that should be set for proper logging. You will need to add your two Application Insights keys here as well. Follow the instructions within the file.

# Application Insights keys and trace configuration
ENV APPINSIGHTS_INSTRUMENTATIONKEY=your_instrumentation_key_goes_here \
    LOCALAPPDATA=/app_insights_data \
    OCAGENT_TRACE_EXPORTER_ENDPOINT=localhost:55678

# The following variables will allow you to filter logs in AppInsights
ENV SERVICE_OWNER=AI4E_Test \
    SERVICE_CLUSTER=Local\ Docker \
    SERVICE_MODEL_NAME=base-py example \
    SERVICE_MODEL_FRAMEWORK=Python \
    SERVICE_MODEL_FRAMEOWRK_VERSION=3.6.6 \
    SERVICE_MODEL_VERSION=1.0

# The API_PREFIX is the URL path that will occur after your domain and before your endpoints
ENV API_PREFIX=/v1/my_api/tasker

You may modify other environment variables as well. In particular, you may want to change the environment variable API_PREFIX. We recommend using the format "/<version-number>/<api-name>/<function>" such as "/v1/my_api/tasker".

(Optional) Set up Azure blob storage

You will want to follow these steps if you are working from the blob-mount-py example. If you do not plan to use Azure blob storage in your app, skip ahead to Build and run your image.
First you will need create a new Azure Blob Container with a file named config.csv. We also recommend using Azure Storage Explorer to aid in storage upload/download.

Create an Azure storage account by selecting "Storage Accounts" from the left menu and clicking the Add button. Make sure to select the resource group you previously created, and use a descriptive name for your storage account (must be lowercase letters or numbers). You may configure advanced options for your account here, or simply click "Review + create".

Create Storage Account

Click "Create" on the validation screen that appears. Once the storage account is deployed, click "Go to resource". You still need to create a container within your storage account. To do this, scroll down on the left menu of your storage account to click on "Blobs". Click the plus sign in the top left to create a new container.

Create Storage Account

Use a text editor to create an empty file named config.csv on your local machine. You can now navigate to your empty Azure container and upload the file as a blob.

Upload config.csv

Next, from within the Azure Portal or within Azure Storage Explorer, copy your blob's storage key. You can find your storage keys by clicking "Keys" on the left menu of your storage account.

Create Storage Account

You must also modify the blob_mount.json file as follows:

Note: You may map as many containers as you would like in this file. The blob mounter will mount all of them.

Build and run your image

This section features a step-by-step guide to building and running your image.

Build your image

  1. Navigate to the directory containing your Dockerfile.
  2. Execute the docker build command:
docker build . -t your_custom_image_name:1

In the above command, -t denotes that you wish to label your image with the name "your_custom_image_name" and with the tag of 1. Typically tags represent the build number.

If you will be pushing your image to a repository, your docker build command will resemble:

docker build . -t your_registry_name.azurecr.io/your_custom_image_name:1

Run your image, locally

Run a container based on your image:

docker run -p 8081:80 "your_custom_image_name:1"

In the above command, the -p switch designates the local port mapping to the container port. -p host_port:container_port. The host_port is arbitrary and will be the port to which you issue requests. Ensure, however, that the container_port is exposed in the Dockerfile with the Dockerfile entry:

EXPOSE 80

TIP: Depending on your git settings and your operating system, the "docker run" command may fail with the error standard_init_linux.go:190: exec user process caused "no such file or directory". If this happens, you need to change the end-of-line characters in startup.sh to LF. One way to do this is using VS Code; open the startup.sh file and click on CRLF in the bottom right corner in the blue bar and select LF instead, then save.

If you find that there are errors and you need to go back and rebuild your docker container, run the following commands:

# This lists all of the docker processes running
docker ps

# Find the container ID in the list from the previous command, and replace <container-id> with that value to end the process
docker kill <container-id>

Then you can execute your docker build and docker run commands again. Additionally, the docker logs are located in your user account's AppData\Local\Docker folder (i.e. C:\Users\jennmar\AppData\Local\Docker).

Make requests

Now that you have a local instance of your container running, you should issue requests and debug it, locally. For this exercise, you may issue requests in whatever way that you would like, but we prefer using Postman to quickly test our endpoints.

Test endpoints

  1. Open Postman or your favorite API development tool.
  2. From your service code, determine which endpoint you want to test. If you are following one of our examples, the endpoint is: http://localhost:<host_port>/<my_api_prefix>/<route>. Also, understand if you will issuing a POST or a GET.
  3. In your API dev tool, select POST or GET and enter the endpoint you would like to test.
  4. If you are performing a POST, construct valid JSON for your endpoint and enter it into the body of the request. Alternatively, if you are POSTing an image, upload it to Postman (see screenshots below).
  5. Click "Send" or execute the call to your endpoint. The running container shell will contain any messages that are printed to the console.

Posting JSON

From the Body tab, select "raw". Ensure that "JSON (application/json)" is selected from the drop-down option (note that a Content-type header is automatically added when you do this). Construct valid JSON in the window.

Post JSON

Posting an Image

In the Headers tab, create a Content-Type header of either image/jpeg or image/png. From the Body tab, select "binary". Upload your JPEG or PNG image here.

Post Image

Post Image

Publish to Azure Container Registry

If you haven't already, create an instance of Azure Container Registry (ACR) in your subscription.

  1. Log into the Azure Portal and click on +Create a Resource. Click on "Containers" and select "Container Registry". Select a name for your registry and make sure you choose the same subscription and resource group you used for the AppInsight step above. Before creating the registry make sure Admin user is set to "enable". Create container registry

  2. After your Azure Container Registry is created, you can click on it to find your login server.

ACR Login Server

Note: if you just created an ACR (i.e., did not include an ACR uri in your previous build), you will need to tag your container image with your ACR uri.

  1. Tag your docker image:
docker tag your_custom_image_name:tag <your_login_server>/your_custom_image_name:tag
  1. Log into your ACR:
docker login --username <username> --password <password> <your_login_server>

You can find your admin ACR credentials on Azure portal.

ACR Credentials

  1. Push your image to your ACR:
docker push <your_login_server>/your_custom_image_name:tag

Run your container in Azure Container Instances

Running your container in ACI is easy. The easiest way is to open the Azure Portal to your ACR, select the repository and tag that corresponds to the image you just pushed, click on it and select "Run Instance." This will create a new ACI instance of your image.

Running Azure Container Instance

Issue requests to your hosted API

Now that your service is running within ACI, we can issue requests to it.

  1. Open the Azure Portal to your ACI. Click on the "Overview" tab and copy the "IP address".
  2. In your API tool, change localhost:port to the IP that you just copied.
  3. Issue your request.

To see logs/console output in your running container, click on "Containers" in your ACI in the Azure Portal and click the "Logs" tab. If you configured Application Insights, you may use that to review logs, identify issues, and view metrics.

Conclusion and Next Steps

Congratulations! You have successfully hosted a model in Azure and exposed it to be accessed as an API.

Cost implications

The services we used today are very reasonably priced. Here are the pricing details.

How to remove Azure resources

We hope you find this a valuable way to provide access to your machine learning model. But if you don't plan to use your API immediately and you want to release these resources in Azure to reduce your costs, you may do so. If you put all resources in a single resource group, then you can navigate to the Azure portal, click on "Resource Groups", and select the resource group that you have been using throughout this tutorial. From there, you can select "Delete resource group" and remove all of the resources at once. (If you didn't add them all to the same resource group, you can delete them all separately.)

Delete Resource Group

Next Steps

Upon completion of this quickstart tutorial, you may want to investigate the following.

FAQs

Contributing

This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.microsoft.com.

When you submit a pull request, a CLA-bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., label, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact opencode@microsoft.com with any additional questions or comments.