microTVM Project API#

About microTVM Project API#

The microTVM Project API allows TVM to automatically run models on unconventional or embedded platforms. It allows platforms to define a standard function to integrate TVM compiler output with boilerplate platform-specific code, producing a runnable Project. Project API then further defines functions to build that project, program compatible devices accessible from the TVM machine, and communicate with the running code so that TVM can perform host-driven inference and autotuning.

There are many cases where it might be desirable simply to invoke microTVM as a tool from your platform’s build process. Indeed, for the average firmware developer, this is likely to be all they need. However, there are a couple of use cases when you may want to teach microTVM how to build firmware using your platform’s build tool:

  1. To enable AutoTVM and AutoScheduling on your platform. Defining a Project API implementation allows TVM to tune models for peak performance on your platform.

  2. To enable engineers without firmware expertise to experiment with models on your platform. Defining a Project API implementation allows these engineers to leverage the standard TVM Python workflows to perform host-driven inference on your platform.

  3. Integration Testing. Defining a Project API implementation allows you to create Continuous Integration Tests which verify model correctness and performance on your platform.

API Definition#

The full API is the abstractmethod defined on ProjectAPIHandler in python/tvm/micro/project_api/server.py. Rather than duplicate the documentation here, we simply refer you to that class.

How TVM uses Project API#

This section explains how the Project API should be used with TVM. Project API is defined around the Project as the buildable unit of firmware. TVM expects to be provided initially with a directory containing a Template Project, which together with a Model Library Format file can be built into a runnable project.

Inside the Template Directory is (typically) a Python script implementing the API server. TVM launches this script in a subprocess and sends commands to the server to perform each of the actions outlined above.

The typical usage flow is as follows:

  1. Launch Project API server in Template Project.

  2. Verify the API server is version-compatible with TVM, plus read properties of the implementation, by sending server_info_query command.

  3. Generate a new project by sending command generate_project to create a new project. The arguments to this command is a Model Library Format and a non-existent directory which should be populated with the generated project. The Template Project API server should copy itself into the newly-generated project.

  4. Terminate the Template Project API server.

  5. Launch Project API server in Generated Project.

  6. Verify the API server is version-compatible with TVM, plus read properties of the implementation, by sending server_info_query command.

  7. Build and flash the projec by sending commands build and flash to the API server.

  8. Communicate with the target. Send command open_transport followed by commands write_transport and read_transport to write and read from e.g. a serial port attached to the target. Upon completion, close_transport is sent.

  9. Terminate Project API server.

Disk Layout of the Project#

In the root directory of a project (template or generated), one of the following two files must exist:

  • microtvm_api_server.py - the suggested approach. Place a python3-compatible Python script in the root directory. TVM will execute this script in its own process using the same interpreter used to execute TVM.

  • microtvm_api_server.sh (on Windows, microtvm_api_server.bat) - alternate approach. When a different Python interpreter is necessary, or when you want to implement the server in a different language, create this executable file. TVM will launch this file in a separate process.

Aside from these two files, no other restrictions are made on the layout.

Communication between TVM and Project API Server#

TVM communicates with the Project API server using JSON-RPC 2.0. TVM always launches API servers using the following command-line:

microtvm_api_server.py --read-fd <n> --write-fd <n>

Commands are sent from TVM to the server over the file descriptor given by --read-fd and replies are received by TVM from the server over the file descriptor given by --write-fd.

Helpers for Implementing the API server in Python#

TVM provides helper utilities that make it easy to implement the server in Python. To implement the server in Python, create microtvm_api_server.py and add from tvm.micro.project_api import server (or, copy this file into your template project–there are no dependencies–and import it there instead). Next, subclass ProjectAPIHander:

class Handler(server.ProjectAPIHandler):
    def server_info_query(self, tvm_version):
        # Implement server_info_query

    def generate_project(self, model_library_format_path, standalone_crt_dir, project_dir, options):
        # Implement generate_project

    # ...

Finally, invoke the helper main():

if __name__ == "__main__":
    server.main(Handler())

Using Project API from tvmc#

Each major Project API command is available through the tvmc micro sub-command to make debugging interactions simple. Invoke tvmc micro --help for more information.