Skip to content

Benchmarks and tunings

Jump to bottom
Thomas Leibovici edited this page Sep 2, 2015 · 4 revisions

Table of Contents

Robinhood Benchmark

Here are a few tips about robinhood benchmarking, indiscriminately:

You can benchmark separately robinhood scanning speed and the database ingestion rate by buildin robinhood with specific flags.

  • to benchmark scans without being impacted by the DB, run ./configure with --enable-bench-scan option.
The main tuning factor in this case is the fs_scan::nb_threads_scan parameter.
  • to benchmark DB ingestion, run ./configure with --enable-bench-db option.
Run the benchmark as a scan. Robinhood will get template entries in the scan directory you provide to generate millions of randomized records to feed the DB.

In this case, the main tunings are in the EntryProcessor block (threads per stage, DB performance strategies: batching, parallelizing...) and of course mysql tunings.

  • to easily script your benchmarks, you can refer to environment variables in rbh config file, eg. nb_threads_scan = $SCAN_THREADS ;
  • Scanning and DB ingestion rate benchmark results are stated in http://lustre.ornl.gov/ecosystem/documents/LustreEco2015-Leibovici.pdf(slides 18 to 23).
  • To benchmark changelog processing, you can generate N records with robinhood shut down, and then run it with --readlog --once options to measure how long its takes to process the accumulated changelogs.

Performance Tuning

  • In ListManager config block, this significantly improves DB performance (but statistics reports take longer):
 user_acct = off
 group_acct = off

It is recommended to switch them off for initial scan.

  • If your scanning is limited by a single client RPC rate, you can distribute it accross multiple clients. See Distributing scans.

Mysql Tuning

Here are classical tunings for mysql/mariadb:

in [mysqld] section: 

 performance_schema
 table_open_cache=2028
 max_connections=1024
 thread_cache_size=512
 table_cache=2048
 connect_timeout=60
 key_buffer_size=512M
 query_cache_size=512M
 query_cache_limit=512M
 sort_buffer_size=512M
 read_rnd_buffer_size=1G
 tmp_table_size=1G
 max_heap_table_size=1G
 
 # innodb_buffer_pool_size: recommended value is 80% of physical memory
 innodb_buffer_pool_size = 100G
 innodb_additional_mem_pool_size = 16M
 innodb_max_dirty_pages_pct = 20
 innodb_file_per_table = 1
 innodb_data_file_path = ibdata1:100M:autoextend
 innodb_write_io_threads = 32
 innodb_read_io_threads = 32
 innodb_flush_method=O_DIRECT
 innodb_io_capacity=100000
 innodb_autoinc_lock_mode = 2
 innodb_thread_concurrency = 0
 innodb_log_buffer_size = 256M
 innodb_log_file_size = 900M
 innodb_log_files_in_group = 4
 innodb_lock_wait_timeout = 120

in [mysqld_safe] section: 

  open-files-limit=2048
  • this script can help for DB tuning:
http://mysqltuner.pl

Linux Tuning

  • check ulimit settings before starting robinhood, especially number of open files & file locks
  • set sysctl vfs_cache pressure for more aggressive reclaim of dentries & inodes

Lustre Client Tuning

  • On your lustre client, check mdc parameters: max_rpcs_in_flight, lru_size & lru_max_age
FYI, its common to tune the lustre osc parameters but robinhood is more sensitive to mdc parameters 
 # example of tunings:
 lctl set_param llite.*.statahead_max=4
 lctl set_param ldlm.namespaces.*.lru_size=100
 lctl set_param ldlm.namespaces.*.lru_max_age=1200
 lctl set_param mdc.*.max_rpcs_in_flight=64
 sysctl -w lnet.debug=0

Back to wiki home

Clone this wiki locally