Each model needs to be measured in the target hardware environment:
- Inference time (ms)
- Energy per inference (mJ), particularly for battery-powered devices
- Power vs. time “P(t)”, peak power, and average power during Inference
- Inference accuracy
System design decisions that can impact these metrics include:
- Chip clock speed and core voltage
- Memory organization, such as: Model and weights in RAM or NVM? Internal or External? Tensor arena stored in Internal or External RAM?
- Vendor supplied Runtimes+SDK
The Solution: ModelCat Hardware Farm
A complete environment for accurate model benchmarking and analysis
- Standardized interface across all vendor hardware
- Connected to a ModelCat Measurement System, which measures inference timing and energy in real time
- REAL results, from REAL HARDWARE
Take a Test Drive