Performance testing

Performance testing

General principles:

• Operations are performed over RAM-allocated arrays (unless specifically targeted to EEPROM)
• Time measured on PC side - therefore communication costs also included. Can be mitigated either by enough repetitions of measured operation (so communication cost is negligible) or two runs with separate number of repetitions are used and subtracted for time computations
• Allocation of new instance of measured algorithm is NOT included in measurement (as good practice is to pre-allocate all objects during applet's install time)

What is measured

• Time necessary to execute separate method (e.g., AESKey.setKey()) [SINGLE_METHOD]
• Time necessary to perform whole cryptographic operation (e.g., transmit X bytes of data to card, encrypt it by AES and send back) are performed. [FULL_OP].
  • TODO: we need to specify testing scenarios - e.g., fixed key, key changed with every apdu...
• Some operations can be measured without additional preparation of target object (e.g., AESKey.setKey()). Target method call is then iterated on-card according to TEST_SETTINGS.numRepeatWholeOperation to suppress of significance of overhead (apdu transmission etc.) on measurement
• Some operations requires some pre-processing (e.g., Cipher.update() requires init() with proper key etc.). This preparation is done in separate command and is not included into measurement.

Results produced (for fixed combination of algorithm, key lengths...)

• Some operations are independent of input data length (e.g., AESKey.setKey()) - presented result is average, min and max value [AMM].
  • _Example (for given key length): AESKey.setKey() takes AVG: 16ms, MIN: 15ms, MAX: 17ms _
• Alternatively, histogram of measured values is produced for operations with high deviations of values from the average (e.g., KeyPair.genKeyPair() for RSA) [HIST].
  • Example: ALG_RSA - KeyPair.genKeyPair():
• Some operations are dependent on input data length (e.g., Cipher.doFinal()) - presented result is graph with X-axis being data length and Y being time required (on single operation) [GDD]. For operations with short underlying blocks (e.g., AES), data lengths tested are 0-256 bytes [GDD256], for operations with bigger underlying blocks 0-512B [GDD512].
  • Example: ALG_DES Cipher.doFinal(): 
• There is overhead associated with making call to measured method (e.g., calling update() once over 256B will take shorter then calling update() four times over 64B parts - common situation when computing MAC over data stored in different arrays). This overhead can significantly impact speed of whole operation. Graph figure is produced with varying number of method calls necessary to process given input data length (for this test, length is fixed at 256B). TEST_SETTINGS.numRepeatSubOperation defines number of measured method invocations necessary to process this fixed amount of data. X-axis shows number of invocations. Y-axis show time required to process 256B (length of data for single operation is 256B / TEST_SETTINGS.numRepeatSubOperation). []

Key classes (AESKey, DESKey...)

• Tested methods: setKey(), getKey(), clearKey() SINGLE_METHOD
• Repeats: whole on-card operation (TEST_SETTINGS.numRepeatWholeOperation)
• Input lengths: not used
• Result: AMM, HIST

Cipher class (ALG_DES_CBC_NOPAD, ALG_RSA_PKCS1...)

• Tested methods: init(), update(), doFinal()
• Repeats: whole on-card operation (TEST_SETTINGS.numRepeatWholeOperation - e.g., repeat of whole encryption with multiple Cipher.update() and one Cipher.doFinal()), measured suboperation repeated (TEST_SETTINGS.numRepeatSubOperation - e.g., Cipher.update())
• Input lengths: multiple lengths according to underlying algorithm block length
• Result: GDD