.. Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file to you under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at .. http://www.apache.org/licenses/LICENSE-2.0 .. Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. ******************************** 基准性能日志格式 ******************************** 此页面详细介绍了统一基准日志格式的模式 v0.1。该模式（schema）将允许与 frameworks/runs, experiment reproduction、data for nightly perf regression 以及日志/可视化工作的分离。 Log 格式概述 ~~~~~~~~~~~~~~~~~~~ 为了简单起见，建议对字段 `workload`、 `engine`、 `hardware` `runtime_ms_mean` 和 `runtime_ms_std` 进行优先级排序。对于更细粒度的日志记录，可以需另外 propagate `*_config` 字段。 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------+------------------------------------------------------------------------------+ | header | examples | category | notes/justification | +=======================+==============================================================================================================================================================================+==============+==============================================================================+ | workload | resnet-18 | workload | name of workload | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------+------------------------------------------------------------------------------+ | engine | "tvm" / "onnxruntime" | compiler | | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------+------------------------------------------------------------------------------+ | hardware | "gcp-c2-standard-16" | hardware | descriptor of target hardware environment | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------+------------------------------------------------------------------------------+ | runtime_ms_mean | 12.452 | statistics | | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------+------------------------------------------------------------------------------+ | runtime_ms_std | 5.3 | statistics | | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------+------------------------------------------------------------------------------+ | timestamp | 1572282699.6 | metadata | indicates when this record is logged | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------+------------------------------------------------------------------------------+ | schema\_version | "0.1" | metadata | ensure reproducibility as we iterate on this schema | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------+------------------------------------------------------------------------------+ | metadata | { "docker\_tag":"gcr.io/.../0a680", ... } | metadata | ``docker_tag`` is optional | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------+------------------------------------------------------------------------------+ | workload\_args | {“input\_name”: "Input3", “input\_shape”: [list\_of\_shape], “data\_layout”: NHCW} | workload | | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------+------------------------------------------------------------------------------+ | workload\_metadata | {"class": "vision","doc\_url": "``https://github.com/.../README.md``", "opset": 7,"type": "body\_analysis","url": "``https://onnxzoo...ferplus.tar.gz``", "md5": "07fc7..."} | workload | source of workload | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------+------------------------------------------------------------------------------+ | engine\_version | "1.0.5" | compiler | use semvar format | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------+------------------------------------------------------------------------------+ | engine\_config | {“llvm”: “llvm-8”, “nvcc”: 10.1, "accelerator": "MLAS", "relay_opt_level": 3, "tvm_target":"llvm -mcpu=cascadelake"} | compiler | fields are optionally specified | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------+------------------------------------------------------------------------------+ | compilation\_config | {"opt_level": 3, "layer_schedules":[]/ } | compiler | fields are optionally specified | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------+------------------------------------------------------------------------------+ | software\_config | {"os": "ubuntu:18.04","pip": { "docker": "4.1.0", "gitpython": "3.0.4", "numpy": "1.17.4", "onnx": "1.6.0"}, “cudnn”: “cudnn-8”, "cuda_driver”: “480.10.1”} | backend | env dependency list | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------+------------------------------------------------------------------------------+ | runtime\_config | {"num_cpu_threads": 3} | backend | info on non-hardware, non-software metadata | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------+------------------------------------------------------------------------------+ | hardware\_config | {"cpu_count": 16, "cloud_machine_type":"c2-standard-16", "memory_GB":64} | hardware | json descriptor of target hardware environment | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------+------------------------------------------------------------------------------+ | execution\_config | {“number”: 1, “repeat”: 10, “min\_repeat\_ms”, 0} | statistics | workload execution parameters | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------+------------------------------------------------------------------------------+ | metrics | {“accuracy”: 48.5,“compilation_ms_mean”: 12} | statistics | other metrics | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------+------------------------------------------------------------------------------+ | runtime_raw | [{"runtime_ms": 12, ...}, {"runtime_ms":13,...},...] | statistics | optional raw metrics array | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------+------------------------------------------------------------------------------+ Storage 格式 ~~~~~~~~~~~~~~ 目前，将基准数据原型化为 JSON 对象，以获得扩展性和便捷性，特别是在该模式的早期版本中。但是，当我们扩展基准聚合和稳定参数时，我们预计会切换到 columnar 格式，如 Arrow 或 Parquet。 下面是 JSON 格式的示例数据： :: { "workload":"arcface_resnet100", "engine":"tvm", "hardware":"gcp-c2-standard-16", "runtime_ms_mean":109.43004820081924, "runtime_ms_std":0.09078385126800587, "timestamp":"20191123003411", "schema_version":"0.1", "metadata":{ "docker_tag":"tlcpack/ci-gpu:v0.53" }, "workload_args":{ "input_shape_dict":{ "data":[ 1, 3, 112, 112 ] }, "input_type_dict":{ "data":"float32" }, "input_value_dict":{} }, "workload_metadata":{ "class":"vision", "doc_url":"https://github.com/onnx/models/blob/main/vision/body_analysis/arcface/README.md", "md5":"66074b860f905295aab5a842be57f37d", "opset":8, "type":"body_analysis", "url":"https://s3.amazonaws.com/onnx-model-zoo/arcface/resnet100/resnet100.tar.gz" }, "engine_version":"1.0.0", "engine_config":{}, "compilation_config":{ "relay_opt_level": 3 }, "software_config":{ "os":"ubuntu:18.04", "pip":{ "docker":"4.1.0", "gitpython":"3.0.4", "numpy":"1.17.4", "onnx":"1.6.0" } }, "runtime_config":{}, "hardware_config":{ "cloud_machine_type":"c2-standard-16", "cloud_provider":"GCP", "cpu_count":16, "cpu_platform":"Intel Cascade Lake", "memory_GB":64 }, "execution_config":{}, "metrics":{} }