Introduction to Core ML conversion tool
Earlier this month, at the WWDC 2017, Apple introduced a ton of new features, including, frameworks and APIs.
One, particularly, catched my attention: Core ML. Core ML lets you integrate trained machine learning models into your app.
The best feature about Core ML is that you don’t need any knowledge about neural networks or machine learning. You can simply use pre-trained models.
A trained model is the result of applying a machine learning algorithm to a set of training data.
— Apple Documentation
There is currently a small set of CoreML models available on Apple’s dedicated website section.
But what if we want to use or create another model? Well, let’s see how!
A real example
In the session 703 (Introducing Core ML), the demo app was a flower classifier, predicting the flower type based on an image of the flower.
I immediately think about PlantNet. It’s a french app that roughly does the same job. However, PlantNet needs an internet connection. The app sends the photo of the flower to their server and you get a list of flowers alongside their match probability.
As I said, you need an internet connection to be able to use the app. But, if like me, you like to wander in the nature, you probably find yourself in a situation where you don’t have cellular data, and so, can’t perform these queries.
That’s where Core ML comes and solves this problem: the model is embeded on the device, so you don’t need anything else to make it work.
Well, let’s try to create our own flower classifier!
The traditional workflow to create a model from scratch can be represented as follows:
The final model is contained in a
.mlmodel file. This is a new open file format that describes the layers in your model, the inputs and outputs, the class labels, and any preprocessing that needs to be done on the data. It also contains all the learned parameters (the weights and biases).
Then we can simply load the
.mlmodel in Xcode and start making predictions!
To makes things easier, let’s assume we already have a Model Source, trained with data. There are ton of this model on the internet. We can focus on the conversion part between the Model Source and the MLModel.
Supported model format
Currently, only the following format are supported for conversion:
The complete (and up to date) list can be found here.
In order to convert our trained model to the
.mlmodel format, Apple released an open source tool, coremltools (the documentation is here).
The last part of the session 710 shows how to use it and convert a basic model. I used it as a base, and I encourage you to watch it.
coremltools requires Python 2.7. Because using Python on macOS is a real pain in the ass, we need to do a little workaround to be able to use
coremltools. Otherwise you may run into meaningless errors (as I did) such as:
Fatal Python error: PyThreadState_Get: no current thread
I assume you already have
pip installed. Let’s details the following commands:
pip install virtualenv virtualenv --python=/usr/bin/python2.7 python27 source python27/bin/activate
Line 1: installs
virtualenv, allowing us to create separate environments, in our case, a Python environment
Line 2: create the Python environment and a directory python27 in the current folder
Line 3: activates the environment
You can now check the current version of Python:
To deactivate this environment, simple use:
Now, in our temporary Python environment, use this to install
pip install -U coremltools
And you’re done! First part achieved 🎉
Now that we have the right tools installed, we need a model to convert! I chose to use a Caffe model, because it’s the one I know the best.
To convert, we need:
.prototxt, describing the
I chose to use a model based on the Oxford 102 category flower dataset.
.caffemodel (~230Mb) can be downloaded here and the
.prototxt can be downloaded here.
You will also need a list of labels, the flower’s name, that can be downloaded here.
Let’s write the script to convert that!
Let’s details the following commands:
import coremltools caffe_model = ('oxford102.caffemodel', 'deploy.prototxt') labels = 'list.txt' coreml_model = coremltools.converters.caffe.convert( caffe_model, class_labels=labels, image_input_names='data' ) coreml_model.author = 'Gael Foppolo' coreml_model.license = 'MIT' coreml_model.short_description = "Guess the flower type" coreml_model.input_description['data'] = 'Input image to be classified' coreml_model.output_description['prob'] = 'Probability of each flower' coreml_model.output_description['classLabel'] = 'Most likely flower on the image' coreml_model.save('FlowerClassifier.mlmodel')
Line 3: defining the Caffe model we would like to convert, using the
Line 7: we use the function provided by
coremltools, using the model defined above ; we also provide the names of the flowers ; finally the name of the input, here an image.
Line 14–21: some customization, to display description in Xcode, when reading the
Line 23: save the model produced to a file
Note: the Caffe model convention for the input’s name is
data, but check the
.prototxt file, looking for the line
input. If the names do not match, the converter will raise an error.
Save this to a
.py file and run it with:
This part can take some times (several minutes), depending on your computer. Grab a coffee or a cup of tea and chill 💅
You should have a
FlowerClassifier.mlmodel file! A working, offline PlantNet application! To try it out, I made a sample app and the results are speaking for themselves:
This sample app and the script to convert the model are available on a GitHub repo! Drag your
FlowerClassifier.mlmodel and your’re good!
Core ML is great for quickly getting a model up and running in your apps. However, there are some limitations due to simplicity:
only supervised machine learning is supported ; no unsupervised or reinforcement
no training on the device
the Core ML conversion tools is limited ; eg, TensorFlow is not supported ; but you can write your own conversion script
Core ML hides its logic ; whether it runs on the CPU or the GPU you don’t know ; and you can’t force it to run on GPU for example
Where to go next?
Now you have a model, it’s up to you use it! For example, using the new Vision framework to feed the model’s input with a live video stream from the camera!