Performance testing - How it works


Measure time (ms) of selected operation running on card (e.g., AESKey.setKey()). Problem: Selected operation cannot be measured directly (e.g., elapsed = timeEnd - timeStart) because of missing on-card timer and protected environment of smart card (no possibility to time start and end of operation inside surrounding code). As a result, one can only measure overall time between send of input data that will trigger operation (PC/SC SCardTransmit()) on host system.

General principles

What is measured

List of tested properties

Performance testing methodology

This measurement then includes time to:

  1. send input to card reader (PC/SC stack)
  2. transmit input data (T=1/T=0)
  3. dispatch command and select on-card method (process())
  4. execute code predeceasing target operation
  5. execute target operation
  6. execute trailing code after target operation
  7. transmit response (dataOut, status word, (T=1/T=0))
  8. receive response (PC/SC stack)

Additionally, target operation usually takes only small fraction of measurement with majority taken by operations we like to exclude from measurement (~one ms vs. ~tens of ms). Situation is additionally worsen by possible non-determistic time fluctuations on a host side.

Note: Measurement fluctuations on host side can be mitigated if simpler host architecture is available (e.g., microcontroller-based card reader).

Note: Very precise measurement of elapsed time can be obtained from a power trace, if start and end of selected operation can be identified. Such a measurement requires access to setup with osciloscope and significant time for identification of target operation. We verified selected operation using this method.

Note: Some operations cannot be meaningfully measured without additional operation executed together (e.g., setKey before target operation Cipher.init). We measure both operations together and then substract time for additional operation.

Note: We intentionally did not exclude outlayer measurements as it may contain interesting information regarding non-deterministic behaviour of a card

Measurement procedure:

  • start/stop time measured on host
  • default length of data for on-card operation (256B)
  • length of variable data for on-card operation (16-512B)
  • repetitions of target operation on card (fixed length => 50x NUM_REPEAT_WHOLE_OPERATION, variable length => 5x (to keep overall running time reasonable))
  • perftest_measure method (outer):
    • check if not already measured before
    • try perftest_measure - catch exception, try again, then ask for user intervention (physical remove of card, reupload of applet)
  • perftest_measure method (innner):
    • for every repetitions of measurement prepare fresh set of objects on card and reset it (prepare_class_XXX, APDU, not measured)
    • Measure processing time without actually calling measured operation (achieved by setting 0 to number of repetitions; testSet.numRepeatWholeOperation set to 0) - repeat 5x NUM_BASELINE_CALIBRATION_RUNS
      • ResetApplet (APDU, not measured)
      • PerfTestCommand
      • => baseline avg time
    • Measure target operation (testSet.numRepeatWholeOperation = 50x fro fixed data or 5x for variable data):
      • ResetApplet (APDU, not measured)
      • PerfTestCommand => time
      • measurement time = time - baseline avg time
      • => min, max, avg measurement time
  • prepare_class_XXX (on-card)
    • allocate new objects required for testing target operation
    • erase or set RAM/EEPROM arrays into default values
  • perftest_class_XXX (on-card)
    • receive apdu data
    • parse incoming settings (num_repeats, data length...)
    • switch (algorithm type)
    • initializations done only once before for(num_repeats)
    • for(num_repeats)
      • target operation
      • (optional) alternate between two different objects to prevent (too quick) use of existing one when card actually decides not to execute operation as the object is still same (e.g., Cipher.init())
    • end, send apdu out


Listed segments show all parts of performance test execution. Part 5 executes target operation we want to measure (green). Since we are not able to perform this execution directly, by subtracting the time of execution of all operations in the left column from all operations in the right column, we gain quite an accurate operation run time. The subtracted time consists of host PC processing (yellow), data transmission between host PC and a card (yellow-blue) and smart card processing (blue).