Hyper-Q Memory Allocator Log File
The memory allocator log file contains metrics about Erlang memory allocation in Hyper-Q. Datometry can use the information provided in the log file to tune memory allocation to improve performance and reliability.
Caution
Datometry provides the following Erlang memory allocation metrics in Hyper-Q for informational purposes. Before modifying any memory allocation parameters, contact Datometry Support to ensure that the changes are compatible with your environment.
The system log is a comma-separated values (CSV) file, a delimited text file that uses a comma to separate values. Each line of the file is a data record. Each record consists of one or more fields, separated by commas. You can open the CSV file in a spreadsheet or comparable tool for analysis.
To learn more more about memory allocation in Erlang, see the following resources:
The Erlang erts_alloc documentation.
Memory Management Battle Stories by Lukas Larsson presented at the Erlang Factory conference. You can download a PDF of the presentation at the following link: http://www.erlang-factory.com/static/upload/media/139454517145429lukaslarsson.pdf
Timestamps and Super Carriers
A carrier is a portion of memory allocated from either the operating system or a super carrier. A super carrier is a large memory area allocated when starting the Erlang virtual machine that can be used during runtime to allocate memory to normal carriers. There are two types of carriers: single-block carriers which only contain one large block, and multi-block carriers which contain multiple blocks.
Super carrier statistics show how much of a super carrier's memory is being used. The values recorded are current when the log entry is written.
To learn more about super carriers, see Super Carrier in the Erlang documentation.
Column | Description |
|---|---|
timestamp | The timestamp when the event occurred, using the local server timezone and formatted with millisecond-precision timestamps as: YYYY-MM-DD HH:MM:SS.mmm |
sc_usage | Percentage of super carrier memory allocated. |
sc_sa | Memory allocated from a super carrier for a super aligned carrier (multi-block carriers). |
sc_sa_usage | Percentage of memory allocated for a super aligned carrier (multi-block carriers). |
sc_sua | Memory allocated from a super carrier for super unaligned carrier (single block carriers). |
sc_sua_usage | Percentage of memory allocated for a super unaligned carrier (single block carriers). |
sc_free_seg_used | Number of free memory segments in a super carrier. |
Log Entries for alloc_util Allocators
The alloc_util framework implements allocators that allocate multi-block carriers and single block carriers from the super carrier or the operating system. The alloc_util allocator statistics show the amount of memory allocated and in use by the allocator.
The statistic names have two parts:
The name of the allocator. For example,
binary_alloc.The statistic for the named allocator. For example,
mbcs_block_count.
To learn more about the allocators, see the Erlang erts_alloc documentation.
In general there is one allocator of a particular type per core. The statistics in the memory allocator log file are a summation of all instances of a given allocator. The values are recent maximum values, meaning the largest value since the last log entry. For the first row, this is the largest value since system startup. This means that values may represent a value at different times between two log entries as values can peak at different times. If the recent maximum value is not available, the current value is used instead. This primarily applies to statistics about multi-block carrier pools.
Column | Description |
|---|---|
mbcs_block_count | The number of memory blocks allocated from multi-block carriers employed by this allocator. |
mbcs_block_size | The total size of memory blocks allocated from multi-block carriers employed by this allocator. |
mbcs_mseg_alloc_carriers | The number of |
mbcs_mseg_alloc_carriers_size | The total memory size of all |
mbcs_sys_alloc_carriers | The number of |
mbcs_sys_alloc_carriers_size | The total size of all |
sbcs_block_count | The number of memory blocks allocated from single block carriers employed by this allocator. |
sbcs_block_size | The total memory size of blocks allocated from single block carriers employed by this allocator for single block carriers. |
sbcs_mseg_alloc_carriers | The number of |
sbcs_mseg_alloc_carriers_size | The total size of all |
sbcs_sys_alloc_carriers | The number of |
sbcs_sys_alloc_carriers_size | The total size of all |
mbcs_pool_block_count | The number of memory blocks allocated from multi-block carriers that are in the migration pool for this allocator. |
mbcs_pool_block_size | The total size of memory blocks allocated from multi-block carriers in the migration pool for this allocator. |
mbcs_pool_carriers | The number of multi-block carriers in the migration pool for this allocator. |
mbcs_pool_carriers_size | Total size of all multi-block carriers in the migration pool for this allocator. |
mbcs_pool_skip_size | Total size of the multi-block carriers that were not examined for block statistics. Hyper-Q skips carrier examination if examining the carrier will cause a block. |
Log Entries for Free Block Fragmentation
For binary_alloc and eheap_alloc, additional information is recorded on the count of free blocks in carriers. These statistics are a summation of all carriers in the allocator.
The statistic names have two parts:
The name of the allocator. For example,
eheap_alloc.The size of the free memory block. For example,
1K_free_block.The free block count is a histogram of the free block sizes in the carrier in increasing sizes. For example,
512_free_blocksis the count of free blocks that are 512 bytes and smaller.1K_free_blocksare memory blocks ranging from 512 bytes to 1 Kbytes.
Column | Description |
|---|---|
unscanned | The number of bytes in the allocator's carriers that were not examined. If the carrier could not be scanned in full without harming the responsiveness of the system, they are skipped. |
size_free_blocks | The number of uncounted blocks size in bytes and smaller in all carriers for the allocator not already included in smaller |