基准性能日志格式

此页面详细介绍了统一基准日志格式的模式 v0.1。该模式（schema）将允许与 frameworks/runs, experiment reproduction、data for nightly perf regression 以及日志/可视化工作的分离。

Log 格式概述

为了简单起见，建议对字段 workload、 engine、 hardware runtime_ms_mean 和 runtime_ms_std 进行优先级排序。对于更细粒度的日志记录，可以需另外 propagate *_config 字段。

header	examples	category	notes/justification
workload	resnet-18	workload	name of workload
engine	“tvm” / “onnxruntime”	compiler
hardware	“gcp-c2-standard-16”	hardware	descriptor of target hardware environment
runtime_ms_mean	12.452	statistics
runtime_ms_std	5.3	statistics
timestamp	1572282699.6	metadata	indicates when this record is logged
schema_version	“0.1”	metadata	ensure reproducibility as we iterate on this schema
metadata	{ “docker_tag”:”gcr.io/…/0a680”, … }	metadata	docker_tag is optional
workload_args	{“input_name”: “Input3”, “input_shape”: [list_of_shape], “data_layout”: NHCW}	workload
workload_metadata	{“class”: “vision”,”doc_url”: “https://github.com/.../README.md”, “opset”: 7,”type”: “body_analysis”,”url”: “https://onnxzoo...ferplus.tar.gz”, “md5”: “07fc7…”}	workload	source of workload
engine_version	“1.0.5”	compiler	use semvar format
engine_config	{“llvm”: “llvm-8”, “nvcc”: 10.1, “accelerator”: “MLAS”, “relay_opt_level”: 3, “tvm_target”:”llvm -mcpu=cascadelake”}	compiler	fields are optionally specified
compilation_config	{“opt_level”: 3, “layer_schedules”:[]/ <SHA_to_schedules>}	compiler	fields are optionally specified
software_config	{“os”: “ubuntu:18.04”,”pip”: { “docker”: “4.1.0”, “gitpython”: “3.0.4”, “numpy”: “1.17.4”, “onnx”: “1.6.0”}, “cudnn”: “cudnn-8”, “cuda_driver”: “480.10.1”}	backend	env dependency list
runtime_config	{“num_cpu_threads”: 3}	backend	info on non-hardware, non-software metadata
hardware_config	{“cpu_count”: 16, “cloud_machine_type”:”c2-standard-16”, “memory_GB”:64}	hardware	json descriptor of target hardware environment
execution_config	{“number”: 1, “repeat”: 10, “min_repeat_ms”, 0}	statistics	workload execution parameters
metrics	{“accuracy”: 48.5,“compilation_ms_mean”: 12}	statistics	other metrics
runtime_raw	[{“runtime_ms”: 12, …}, {“runtime_ms”:13,…},…]	statistics	optional raw metrics array

Storage 格式

目前，将基准数据原型化为 JSON 对象，以获得扩展性和便捷性，特别是在该模式的早期版本中。但是，当我们扩展基准聚合和稳定参数时，我们预计会切换到 columnar 格式，如 Arrow 或 Parquet。

下面是 JSON 格式的示例数据：

{
  "workload":"arcface_resnet100",
  "engine":"tvm",
  "hardware":"gcp-c2-standard-16",
  "runtime_ms_mean":109.43004820081924,
  "runtime_ms_std":0.09078385126800587,
  "timestamp":"20191123003411",
  "schema_version":"0.1",
  "metadata":{
    "docker_tag":"tlcpack/ci-gpu:v0.53"
  },
  "workload_args":{
    "input_shape_dict":{
      "data":[
        1,
        3,
        112,
        112
      ]
    },
    "input_type_dict":{
      "data":"float32"
    },
    "input_value_dict":{}
  },
  "workload_metadata":{
    "class":"vision",
    "doc_url":"https://github.com/onnx/models/blob/main/vision/body_analysis/arcface/README.md",
    "md5":"66074b860f905295aab5a842be57f37d",
    "opset":8,
    "type":"body_analysis",
    "url":"https://s3.amazonaws.com/onnx-model-zoo/arcface/resnet100/resnet100.tar.gz"
  },
  "engine_version":"1.0.0",
  "engine_config":{},
  "compilation_config":{
    "relay_opt_level": 3
  },
  "software_config":{
    "os":"ubuntu:18.04",
    "pip":{
      "docker":"4.1.0",
      "gitpython":"3.0.4",
      "numpy":"1.17.4",
      "onnx":"1.6.0"
    }
  },
  "runtime_config":{},
  "hardware_config":{
    "cloud_machine_type":"c2-standard-16",
    "cloud_provider":"GCP",
    "cpu_count":16,
    "cpu_platform":"Intel Cascade Lake",
    "memory_GB":64
  },
  "execution_config":{},
  "metrics":{}
}