## taosBenchmark ### Introduction taosBenchmark is an application for TDengine's performance testing. taosBenchmark can test TDengine's insertion, query and subscription functionalities, it can simulate massive data generated by a large number of devices and also flexibly control the number of columns, data types, threads and so on. formerly called taosdemo, now renamed to taosBenchmark, and the installation package provides taosdemo as a soft link to taosBenchmark. taosBenchmark support two kinds of configuration, one is command line and the other is [json file](#taosbenchmarkjson-configuration-file). ### taosBenchmark CLI Options | Option Name | Description | | :--------------------------------------------------- | ------------------------------------------------------------ | | [-f/--file](#taosbenchmarkjson-configuration-file) | json configuration file | | -c/--config-dir | The directory where the config file in, the default path is /etc/taos/ | | -h/--host | FQDN used to connect taosd server, the default value is localhost | | -P/--port | port number used to connect taosd server, the default value is 6030 | | -I/--interface | how taosBenchmark insert data, the options are taosc, rest, stmt, sml, the default value is taosc | | -u/--user | user name used to connect taosd server, the default value is root | | -p/--password | password used to connect taosd server, the default value is taosdata | | -o/--ouput | specify the path of the result output file, the default value is ./output.txt | | -T/--threads | specify the number of thread to insert data, the default value is 8 | | [-i/--insert-interval](#-i--insert-interval) | the insert interval for interlace insert mode, unit is ms, default value is 0 | | -S/--time-step | insert timestamp step for each record in each sub-table, unit is ms, the default value is 1 | | [-B/--interlace-rows](#-b--interlace-rows) | the number of interlaced row insert into child tables | | -r/--rec-per-req | the number of records in each insert request, the default value is 30000 | | -t/--tables | the number of child tables, the default value is 10000 | | -n/--records | the number of record inserted for each child tables, the default value is 10000 | | -d/--database | the name of the database, the default value is test | | [-l/--columns](#-l--columns) | the number of columns in child tables, will use columns with int data type. | | [-A/--tag-type](#-a--tag-type) | the data type of child tables' tags | | [-b/--data-type](#-b--data-type) | the data type of child tables' columns | | -w/--binwidth | the default length of nchar and binary data type, the default value is 64 | | -m/--table-prefix | the prefix of child tables' name, the default value is d | | -E/--escape-character | use escape character in stable and sub-table name, optional. | | -C/--chinese | nchar and binary are basic unicode chinese characters, optional. | | [-N/--normal-table](#-n--normal-table) | only create normal table without super table, optional | | -M/--random | the data source is random, optional | | -x/--aggr-func | query aggregation function after insertion, optional | | -y/--answer-yes | pass confirmation prompt to continue, optional | | [-R/--disorder-range](#-r--disorder-range) | the range of disorder timestamp, based on database's precision, the default value is 1000 | | [-O/--disorder](#-o--disorder) | the ratio of inserting data with disorder timestamp, the default is 0 | | -a/--replica | the number of replica when create the database, the default value is 1 | | -V/--version | display version information and exit | | -?/--help | display help message and exit | ### taosBenchmark json configuration file #### 1、insert json configuration file ```json { "filetype": "insert", "cfgdir": "/etc/taos", "host": "127.0.0.1", "port": 6030, "user": "root", "password": "taosdata", "thread_count": 4, "result_file": "./insert_res.txt", "confirm_parameter_prompt": "no", "insert_interval": 0, "interlace_rows": 100, "num_of_records_per_req": 100, "prepared_rand": 10000, "chinese":"no", "databases": [{ "dbinfo": { "name": "db", "drop": "yes", "replica": 1, "days": 10, "cache": 16, "blocks": 8, "precision": "ms", "keep": 3650, "minRows": 100, "maxRows": 4096, "comp":2, "walLevel":1, "cachelast":0, "quorum":1, "fsync":3000, "update": 0 }, "super_tables": [{ "name": "stb", "child_table_exists":"no", "childtable_count": 100, "childtable_prefix": "stb_", "escape_character": "yes", "batch_create_tbl_num": 5, "data_source": "rand", "insert_mode": "taosc", "line_protocol": "line", "insert_rows": 100000, "childtable_limit": 10, "childtable_offset":100, "interlace_rows": 0, "insert_interval":0, "disorder_ratio": 0, "disorder_range": 1000, "timestamp_step": 10, "start_timestamp": "2020-10-01 00:00:00.000", "sample_format": "csv", "sample_file": "./sample.csv", "use_sameple_ts": "no", "tags_file": "", "columns": [{"type": "INT"}, {"type": "DOUBLE", "count":10}, {"type": "BINARY", "len": 16, "count":3}, {"type": "BINARY", "len": 32, "count":6}], "tags": [{"type": "TINYINT", "count":2}, {"type": "BINARY", "len": 16, "count":5}] }] }] } ``` #### Parameters: | Group | Option Name | Description | | ------------ | --------------------------------------- | ------------------------------------------------------------ | | | filetype | file type to specify which kind of test, for insertion test, required insert | | | cfgdir | the directory where taos configuration file in, the default value is /etc/taos | | | host | FQDN of taosd server, default is localhost | | | port | port number of taosd server, default is 6030 | | | user | username to connect taosd server, default is root | | | password | password to connect taosd server, default is taosdata | | | thread_count | the number of thread to insert and create tables, default is 8 | | | result_file | file path to save the output result, default is ./output.txt | | | confirm_parameter_prompt | pass the confirmation prompt during execution, the default is no | | | [insert_interval](#-i--insert-interval) | insert interval for interlace mode, the default value is 0 | | | [interlace_rows](#-b--interlace-rows) | the number of interlace row for each sub-table, the default value is 0 | | | num_of_records_per_req | the number of record in each request, the default is 30000 | | | [prepared_rand](#prepare_rand) | the number of randomly generated data, the default value is 10000 | | | chinese | nchar and binary are rand chinese, the default value is no | | dbinfo | name | database name, required | | dbinfo | drop | whether drop database before insert test, the default value is yes | | dbinfo | replica | number of replica, default value is 1 | | dbinfo | days | time span for storing data in file, default value is 10 | | dbinfo | cache | size of memory block, unit is MB, default value is 16 | | dbinfo | blocks | the number of cache-sized memory block in each vnode(tsdb), default value is 6 | | dbinfo | precision | database time precision, default value is "ms" | | dbinfo | keep | number of day for reserving data, default value is 3650 | | dbinfo | minRows | minimum number of records in a file block, default value is 100 | | dbinfo | maxRows | maximum number of records in a file block, default value is 4096 | | dbinfo | comp | file compression flag, default value is 2 | | dbinfo | walLevel | wal level, default value is 1 | | dbinfo | cachelast | whether allow the last record of each table to be kept in memory, default is 0 | | dbinfo | quorum | number of acknowledgement required for asynchronous write, default is 1 | | dbinfo | fsync | interval for fsync when wal is set to 2, unit is ms, default value is 3000 | | dbinfo | update | wheter support data update, default value is 0 | | super_tables | name | name of super table, required | | super_tables | child_table_exists | whether child table already exists, default is no | | super_tables | childtable_count | number of child table, required | | super_tables | childtable_prefix | prefix of child table name, required | | super_tables | escape_character | super table and child table name include escape character, default is no | | super_tables | batch_create_tbl_num | number of child table created for each request, default is 10 | | super_tables | data_source | data resource type | | super_tables | insert_mode | insert mode, options: taosc, rest, stmt, sml, default is taosc | | super_tables | line_protocol | valid only when insert_mode is sml, options: line, telnet, json, default is line | | super_tables | insert_rows | number of record for each child table, default is 0 | | super_tables | childtable_offset | offset of child table to insert data, valid only when drop is no and child_table_exists is yes | | super_tables | childtable_limit | number of child table to insert data, valid only when drop is no and child_table_exists is yes | | super_tables | interlace_rows | interlace row for each child table, default is 0 | | super_tables | insert_interval | insert interval between two request, valid when interlace_rows larger than 0 | | super_tables | [disorder_ratio](#-r--disorder-ratio) | percentage of data with disorder timestamp, default is 0 | | super_tables | [disorder_range](#-r--disorder-range) | range of disorder timestamp, valid only when disorder_ratio is larger than 0, default is 1000 | | super_tables | timestamp_step | timestamp step for each record, default is 1 | | super_tables | start_timestamp | start value of timestamp for each child table, default value is now | | super_tables | sample_format | type of sample data file, only support csv now | | super_tables | sample_file | sample file, valid only when sample_source is "sample" | | super_tables | use_sample_ts | whether the sample file include timestamp, default is no | | super_tables | tags_file | tag data sample file, only support in taosc, rest insert mode | | columns/tags | type | data type, required | | columns/tags | len | length of data, valid for nchar and binary, default is 8 | | columns/tags | count | number of consecutive of this column, default is 1 | #### 2、Query test json configuration file ```json { "filetype": "query", "cfgdir": "/etc/taos", "host": "127.0.0.1", "port": 6030, "user": "root", "password": "taosdata", "confirm_parameter_prompt": "no", "databases": "db", "query_times": 2, "query_mode": "taosc", "specified_table_query": { "query_interval": 1, "concurrent": 3, "sqls": [ { "sql": "select last_row(*) from stb0 ", "result": "./query_res0.txt" }, { "sql": "select count(*) from stb00_1", "result": "./query_res1.txt" } ] }, "super_table_query": { "stblname": "stb1", "query_interval": 1, "threads": 3, "sqls": [ { "sql": "select last_row(ts) from xxxx", "result": "./query_res2.txt" } ] } } ``` #### Parameters: | Group | Options | Description | | --------------------------------------- | ------------------------ | ------------------------------------------------------------ | | | filetype | file type to specify which kind of test, for query test, required | | | cfgdir | the directory where taos configuration file in | | | host | FQDN of taosd server, default is localhost | | | port | port number of taosd server, default is 6030 | | | user | username to connect taosd server, default is root | | | password | password to connect taosd server, default is taosdata | | | confirm_parameter_prompt | pass the confirmation prompt during execution, the default is no | | | databases | name of database, required | | | query_times | number of query times | | | query_mode | query mode, options: taosc and rest, default is taosc | | specified_table_query/super_table_query | query_interval | query interval, unit is second, default is 0 | | specified_table_query/super_table_query | concurrent/threads | number of thread to execute sql, default is 1 | | super_table_query | stblname | supertable name, required | | sqls | [sql](#sql) | sql command, required | | sqls | result | result file for query result, empty for none | #### 3、Subscribe json configuraion file ```json { "filetype":"subscribe", "cfgdir": "/etc/taos", "host": "127.0.0.1", "port": 6030, "user": "root", "password": "taosdata", "databases": "db", "confirm_parameter_prompt": "no", "specified_table_query": { "concurrent":1, "interval":0, "restart":"yes", "keepProgress":"yes", "sqls": [ { "sql": "select * from stb00_0 ;", "result": "./subscribe_res0.txt" } ] }, "super_table_query": { "stblname": "stb0", "threads":1, "interval":10000, "restart":"yes", "keepProgress":"yes", "sqls": [ { "sql": "select * from xxxx where ts > '2021-02-25 11:35:00.000' ;", "result": "./subscribe_res1.txt" }] } } ``` #### Parameters： | Group | Options | Description | | --------------------------------------- | ------------------------ | ------------------------------------------------------------ | | | filetype | file type to specify which kind of test, for subsribe test, required | | | cfgdir | the directory of taos configuration file | | | host | FQDN of taosd server, default is localhost | | | port | port number of taosd server, default is 6030 | | | user | username to connect taosd server, default is root | | | password | password to connect taosd server, default is taosdata | | | databases | database name, required | | | confirm_parameter_prompt | whether pass the confirmation prompt during execution | | specified_table_query/super_table_query | concurrent/threads | number of thread to execute sqls, default is 1 | | specified_table_query/super_table_query | interval | interval to execute subscribe, default is 0 | | specified_table_query/super_table_query | restart | no: continue previous subscription, yes: start a new subscription | | specified_table_query/super_table_query | keepProgress | whether keep the subscribe progress | | specified_table_query/super_table_query | resubAfterConsume | whether unsubscribe and then subscribe again | | super_table_query | stblname | name of supertable, required | | sqls | [sql](#sql) | sql command, required | | sqls | result | result file for query result, empty for none | - ##### -i/--insert-interval Only work when interlace rows(-B/--interlace-rows) is larger than zero. The interval means thread will sleep that much of time after it inserted interlace rows for each child table assign to it. - ##### -B/--interlace-rows If its value is zero, means insert progressively, thread will insert into child table one by one. If its larger than zero, thread will first insert into that number of rows to the first child table, then second, third and so on, after all child table have been inserted that number of rows, thread will restart insert from the first child table and in turn. - ##### -l/--columns if both -l/--columns and -b/--data-type is set, it will check whether the column number is reach to -l/--columns for -b/--data-type, if it is yes, ignore this option, or it will continue add columns to reach this number with all int data type. - ##### -A/--tag-type set the tag type for super table, nchar and binary can also set length, such as ``` taosBenchmark -A INT,DOUBLE,NCHAR,BINARY(16) ``` default is INT,BINARY(16) - ##### -b/--data-type same as -A/--tag-type, but for columns, and default is FLOAT,INT,FLOAT - ##### -O/--disorder the ratio of disorder timestamp, maximum is 50 - ##### -R/--disorder-range valid only if -O/--disorder is larger than zero, and the disorder means timestamp will decrease random number of millisecond with that range when generated. - ##### prepared_rand the number of random data pre-generate as data source, small prepared_rand will save memory but reduce data variety. - ##### sql for super table query sql, keep "xxxx" in the sql command, program will auto replace it with all child table name in super table