Skip to main content

Datometry Documentation

Hyper-Q Memory Allocator Log File

The memory allocator log file contains metrics about Erlang memory allocation in Hyper-Q. Datometry can use the information provided in the log file to tune memory allocation to improve performance and reliability.

Caution

Datometry provides the following Erlang memory allocation metrics in Hyper-Q for informational purposes. Before modifying any memory allocation parameters, contact Datometry Support to ensure that the changes are compatible with your environment.

The system log is a comma-separated values (CSV) file, a delimited text file that uses a comma to separate values. Each line of the file is a data record. Each record consists of one or more fields, separated by commas. You can open the CSV file in a spreadsheet or comparable tool for analysis.

To learn more more about memory allocation in Erlang, see the following resources:

Timestamps and Super Carriers

A carrier is a portion of memory allocated from either the operating system or a super carrier. A super carrier is a large memory area allocated when starting the Erlang virtual machine that can be used during runtime to allocate memory to normal carriers. There are two types of carriers: single-block carriers which only contain one large block, and multi-block carriers which contain multiple blocks.

Super carrier statistics show how much of a super carrier's memory is being used. The values recorded are current when the log entry is written.

To learn more about super carriers, see Super Carrier in the Erlang documentation.

Table 16. Hyper-Q Memory Allocator Log File Columns for Super Carriers

Column

Description

timestamp

The timestamp when the event occurred, using the local server timezone and formatted with millisecond-precision timestamps as: YYYY-MM-DD HH:MM:SS.mmm

sc_usage

Percentage of super carrier memory allocated.

sc_sa

Memory allocated from a super carrier for a super aligned carrier (multi-block carriers).

sc_sa_usage

Percentage of memory allocated for a super aligned carrier (multi-block carriers).

sc_sua

Memory allocated from a super carrier for super unaligned carrier (single block carriers).

sc_sua_usage

Percentage of memory allocated for a super unaligned carrier (single block carriers).

sc_free_seg_used

Number of free memory segments in a super carrier.



Log Entries for alloc_util Allocators

The alloc_util framework implements allocators that allocate multi-block carriers and single block carriers from the super carrier or the operating system. The alloc_util allocator statistics show the amount of memory allocated and in use by the allocator.

The statistic names have two parts:

  • The name of the allocator. For example, binary_alloc.

  • The statistic for the named allocator. For example, mbcs_block_count.

To learn more about the allocators, see the Erlang erts_alloc documentation.

In general there is one allocator of a particular type per core. The statistics in the memory allocator log file are a summation of all instances of a given allocator. The values are recent maximum values, meaning the largest value since the last log entry. For the first row, this is the largest value since system startup. This means that values may represent a value at different times between two log entries as values can peak at different times. If the recent maximum value is not available, the current value is used instead. This primarily applies to statistics about multi-block carrier pools.

Table 17. Hyper-Q Memory Allocator Log File Columns for Single and Multi-Block Carriers

Column

Description

mbcs_block_count

The number of memory blocks allocated from multi-block carriers employed by this allocator.

mbcs_block_size

The total size of memory blocks allocated from multi-block carriers employed by this allocator.

mbcs_mseg_alloc_carriers

The number of mseg_alloc carriers employed by this allocator for multi-block carriers.

mbcs_mseg_alloc_carriers_size

The total memory size of all mseg_alloc carriers employed by this allocator for multi-block carriers.

mbcs_sys_alloc_carriers

The number of sys_alloc carriers employed by this allocator for multi-block carriers.

mbcs_sys_alloc_carriers_size

The total size of all sys_alloc carriers employed by this allocator for multi-block carriers.

sbcs_block_count

The number of memory blocks allocated from single block carriers employed by this allocator.

sbcs_block_size

The total memory size of blocks allocated from single block carriers employed by this allocator for single block carriers.

sbcs_mseg_alloc_carriers

The number of mseg_alloc carriers employed by this allocator for single block carriers.

sbcs_mseg_alloc_carriers_size

The total size of all mseg_alloc carriers employed by this allocator for single block carriers.

sbcs_sys_alloc_carriers

The number of sys_alloc carriers employed by this allocator for single block carriers.

sbcs_sys_alloc_carriers_size

The total size of all sys_alloc carriers employed by this allocator for single block carriers.

mbcs_pool_block_count

The number of memory blocks allocated from multi-block carriers that are in the migration pool for this allocator.

mbcs_pool_block_size

The total size of memory blocks allocated from multi-block carriers in the migration pool for this allocator.

mbcs_pool_carriers

The number of multi-block carriers in the migration pool for this allocator.

mbcs_pool_carriers_size

Total size of all multi-block carriers in the migration pool for this allocator.

mbcs_pool_skip_size

Total size of the multi-block carriers that were not examined for block statistics. Hyper-Q skips carrier examination if examining the carrier will cause a block.



Log Entries for Free Block Fragmentation

For binary_alloc and eheap_alloc, additional information is recorded on the count of free blocks in carriers. These statistics are a summation of all carriers in the allocator.

The statistic names have two parts:

  • The name of the allocator. For example, eheap_alloc.

  • The size of the free memory block. For example, 1K_free_block.

    The free block count is a histogram of the free block sizes in the carrier in increasing sizes. For example, 512_free_blocks is the count of free blocks that are 512 bytes and smaller. 1K_free_blocks are memory blocks ranging from 512 bytes to 1 Kbytes.

Table 18. Hyper-Q Memory Allocator Log File Columns for Free Block Fragmentation

Column

Description

unscanned

The number of bytes in the allocator's carriers that were not examined. If the carrier could not be scanned in full without harming the responsiveness of the system, they are skipped.

size_free_blocks

The number of uncounted blocks size in bytes and smaller in all carriers for the allocator not already included in smaller size_free_blocks values. The largest size_free_blocks includes all remaining blocks regardless of size.