..  Licensed to the Apache Software Foundation (ASF) under one
    or more contributor license agreements.  See the NOTICE file
    distributed with this work for additional information
    regarding copyright ownership.  The ASF licenses this file
    to you under the Apache License, Version 2.0 (the
    "License"); you may not use this file except in compliance
    with the License.  You may obtain a copy of the License at

..    http://www.apache.org/licenses/LICENSE-2.0

..  Unless required by applicable law or agreed to in writing,
    software distributed under the License is distributed on an
    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    KIND, either express or implied.  See the License for the
    specific language governing permissions and limitations
    under the License.

********************************
基准性能日志格式
********************************

此页面详细介绍了统一基准日志格式的模式 v0.1。该模式(schema)将允许与 frameworks/runs, experiment reproduction、data for nightly perf regression 以及日志/可视化工作的分离。

Log 格式概述
~~~~~~~~~~~~~~~~~~~

为了简单起见,建议对字段 `workload`、 `engine`、 `hardware` `runtime_ms_mean` 和 `runtime_ms_std` 进行优先级排序。对于更细粒度的日志记录,可以需另外 propagate `*_config` 字段。

+-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------+------------------------------------------------------------------------------+
| header                | examples                                                                                                                                                                     | category     | notes/justification                                                          |
+=======================+==============================================================================================================================================================================+==============+==============================================================================+
| workload              | resnet-18                                                                                                                                                                    | workload     | name of workload                                                             |
+-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------+------------------------------------------------------------------------------+
| engine                | "tvm" / "onnxruntime"                                                                                                                                                        | compiler     |                                                                              |
+-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------+------------------------------------------------------------------------------+
| hardware              | "gcp-c2-standard-16"                                                                                                                                                         | hardware     | descriptor of target hardware environment                                    |
+-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------+------------------------------------------------------------------------------+
| runtime_ms_mean       | 12.452                                                                                                                                                                       | statistics   |                                                                              |
+-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------+------------------------------------------------------------------------------+
| runtime_ms_std        | 5.3                                                                                                                                                                          | statistics   |                                                                              |
+-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------+------------------------------------------------------------------------------+
| timestamp             | 1572282699.6                                                                                                                                                                 | metadata     | indicates when this record is logged                                         |
+-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------+------------------------------------------------------------------------------+
| schema\_version       | "0.1"                                                                                                                                                                        | metadata     | ensure reproducibility as we iterate on this schema                          |
+-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------+------------------------------------------------------------------------------+
| metadata              | { "docker\_tag":"gcr.io/.../0a680", ... }                                                                                                                                    | metadata     | ``docker_tag`` is optional                                                   |
+-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------+------------------------------------------------------------------------------+
| workload\_args        | {“input\_name”: "Input3", “input\_shape”: [list\_of\_shape], “data\_layout”: NHCW}                                                                                           | workload     |                                                                              |
+-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------+------------------------------------------------------------------------------+
| workload\_metadata    | {"class": "vision","doc\_url": "``https://github.com/.../README.md``", "opset": 7,"type": "body\_analysis","url": "``https://onnxzoo...ferplus.tar.gz``", "md5": "07fc7..."} | workload     | source of workload                                                           |
+-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------+------------------------------------------------------------------------------+
| engine\_version       | "1.0.5"                                                                                                                                                                      | compiler     | use semvar format                                                            |
+-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------+------------------------------------------------------------------------------+
| engine\_config        | {“llvm”: “llvm-8”, “nvcc”: 10.1, "accelerator": "MLAS", "relay_opt_level": 3, "tvm_target":"llvm -mcpu=cascadelake"}                                                         | compiler     | fields are optionally specified                                              |
+-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------+------------------------------------------------------------------------------+
| compilation\_config   | {"opt_level": 3, "layer_schedules":[]/ <SHA_to_schedules>}                                                                                                                   | compiler     | fields are optionally specified                                              |
+-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------+------------------------------------------------------------------------------+
| software\_config      | {"os": "ubuntu:18.04","pip": { "docker": "4.1.0", "gitpython": "3.0.4", "numpy": "1.17.4", "onnx": "1.6.0"}, “cudnn”: “cudnn-8”, "cuda_driver”: “480.10.1”}                  | backend      | env dependency list                                                          |
+-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------+------------------------------------------------------------------------------+
| runtime\_config       | {"num_cpu_threads": 3}                                                                                                                                                       | backend      | info on non-hardware, non-software metadata                                  |
+-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------+------------------------------------------------------------------------------+
| hardware\_config      | {"cpu_count": 16, "cloud_machine_type":"c2-standard-16", "memory_GB":64}                                                                                                     | hardware     | json descriptor of target hardware environment                               |
+-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------+------------------------------------------------------------------------------+
| execution\_config     | {“number”: 1, “repeat”: 10, “min\_repeat\_ms”, 0}                                                                                                                            | statistics   | workload execution parameters                                                |
+-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------+------------------------------------------------------------------------------+
| metrics               | {“accuracy”: 48.5,“compilation_ms_mean”: 12}                                                                                                                                 | statistics   | other metrics                                                                |
+-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------+------------------------------------------------------------------------------+
| runtime_raw           | [{"runtime_ms": 12, ...}, {"runtime_ms":13,...},...]                                                                                                                         | statistics   | optional raw metrics array                                                   |
+-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------+------------------------------------------------------------------------------+



Storage 格式
~~~~~~~~~~~~~~

目前,将基准数据原型化为 JSON 对象,以获得扩展性和便捷性,特别是在该模式的早期版本中。但是,当我们扩展基准聚合和稳定参数时,我们预计会切换到 columnar 格式,如 Arrow 或 Parquet。

下面是 JSON 格式的示例数据:

::

  {
    "workload":"arcface_resnet100",
    "engine":"tvm",
    "hardware":"gcp-c2-standard-16",
    "runtime_ms_mean":109.43004820081924,
    "runtime_ms_std":0.09078385126800587,
    "timestamp":"20191123003411",
    "schema_version":"0.1",
    "metadata":{
      "docker_tag":"tlcpack/ci-gpu:v0.53"
    },
    "workload_args":{
      "input_shape_dict":{
        "data":[
          1,
          3,
          112,
          112
        ]
      },
      "input_type_dict":{
        "data":"float32"
      },
      "input_value_dict":{}
    },
    "workload_metadata":{
      "class":"vision",
      "doc_url":"https://github.com/onnx/models/blob/main/vision/body_analysis/arcface/README.md",
      "md5":"66074b860f905295aab5a842be57f37d",
      "opset":8,
      "type":"body_analysis",
      "url":"https://s3.amazonaws.com/onnx-model-zoo/arcface/resnet100/resnet100.tar.gz"
    },
    "engine_version":"1.0.0",
    "engine_config":{},
    "compilation_config":{
      "relay_opt_level": 3
    },
    "software_config":{
      "os":"ubuntu:18.04",
      "pip":{
        "docker":"4.1.0",
        "gitpython":"3.0.4",
        "numpy":"1.17.4",
        "onnx":"1.6.0"
      }
    },
    "runtime_config":{},
    "hardware_config":{
      "cloud_machine_type":"c2-standard-16",
      "cloud_provider":"GCP",
      "cpu_count":16,
      "cpu_platform":"Intel Cascade Lake",
      "memory_GB":64
    },
    "execution_config":{},
    "metrics":{}
  }