Blog

Atom Feed

Mixed Memories

Why do macOS programs show different amounts of used memory?

9 Jul 2023

For most Mac users, determining the amount of memory used is pretty simple. You go into Activity Monitor and have a look at the Memory tab.

Activity Monitor even gives us different subcategories of memory usage, which is nice. But as we will see soon, very simplified.

The more technically inclined might go one step further, and use something like iStat Menus or neofetch. I'm too cheap to buy a license for the former, so let's have a look at the latter.

Looks like memory to me.

But they'll quickly notice a small problem: the amount of used memory can sometimes be different.1 And while in this case, one is in mebibytes and the other in megabytes,2 the difference still doesn't line up when you do the conversion. So let's go one step lower and query the amount of memory by compiling a little executable in a trendy programming language such as Rust.

use sysinfo::{System, SystemExt};

fn main() -> () {
    let sys = System::new_all();

    print!(
        "{mem_used} bytes / {mem} bytes",
        mem_used = sys.used_memory(),
        mem = sys.total_memory()
    );
}

Compiling test_crate v0.0.1 (file path redacted)
    Finished dev [unoptimized + debuginfo] target(s) in 0.26s
    Running `target/debug/test_crate`
15674580992 bytes / 17179869184 bytes

Well, that isn't helpful. Let's actually have a look at going on here. Unfortunately, Activity Monitor isn't open source,3 but we can have a look at the internals of the neofetch source code and the sysinfo crate I used. I'm going to start with the former.

At its core, Neofetch is a one-file Bash script that handles all its functionality for a variety of (largely Unix-like, but also Windows) operating systems. It is divided into different functions, each of which is used to obtain the appropriate statistics for the respective category. We want the get_memory() function. It's divided into cases for different operating systems, so we'll grab the section of code that gets the macOS info.

"Mac OS X" | "macOS" | "iPhone OS")
    mem_total="$(($(sysctl -n hw.memsize) / 1024 / 1024))"
    mem_wired="$(vm_stat | awk '/ wired/ { print $4 }')"
    mem_active="$(vm_stat | awk '/ active/ { printf $3 }')"
    mem_compressed="$(vm_stat | awk '/ occupied/ { printf $5 }')"
    mem_compressed="${mem_compressed:-0}"
    mem_used="$(((${mem_wired//.} + ${mem_active//.} + ${mem_compressed//.}) * 4 / 1024))"
;;

As one can see, in typical Unix fashion, it calls a number of small utilities to gather the amount of used memory and total memory. In this function, they are sysctl, which is a command-line tool that allows querying system statistics, and vm_stat, which lists off various Mach kernel4 memory statistics.

First, it grabs the grabs the total memory and divides it twice to convert it into mebibytes. It then pulls together wired memory, active memory, and compressed memory to compute the used memory. Since the figures it uses are in pages, it then uses the page size5 to convert these pages into bytes, and then divides it as it did with the total memory.

Now, let's look at the Rust sysinfo crate I used earlier:

Much of the library is composed of OS-specific code that lives in its respective folder. In our case, that's src/apple, for macOS and iOS code. In particular, we want the functions under SystemExt, so we'll grab src/apple/system.rs.

// get ram info
if self.mem_total < 1 {
    get_sys_value(
        libc::CTL_HW as _,
        libc::HW_MEMSIZE as _,
        mem::size_of::<u64>(),
        &mut self.mem_total as *mut u64 as *mut c_void,
        &mut mib,
    );
}
let mut count: u32 = libc::HOST_VM_INFO64_COUNT as _;
let mut stat = mem::zeroed::<vm_statistics64>();
if host_statistics64(
    self.port,
    libc::HOST_VM_INFO64,
    &mut stat as *mut vm_statistics64 as *mut _,
    &mut count,
) == libc::KERN_SUCCESS
{
    // From the apple documentation:
    //
    // /*
    //  * NB: speculative pages are already accounted for in "free_count",
    //  * so "speculative_count" is the number of "free" pages that are
    //  * used to hold data that was read speculatively from disk but
    //  * haven't actually been used by anyone so far.
    //  */
    self.mem_available = u64::from(stat.free_count)
        .saturating_add(u64::from(stat.inactive_count))
        .saturating_mul(self.page_size_kb);
    self.mem_free = u64::from(stat.free_count)
        .saturating_sub(u64::from(stat.speculative_count))
        .saturating_mul(self.page_size_kb);
}

Further down, the used memory is computed by subtracting mem_free from mem_total. Makes sense, I guess. As for the calculations themselves, they are obtained via macOS system calls, which is more efficient than launching a number of Unix command-line processes. The total memory is retreived using the same system call as Neofetch (hw.memsize), but it uses the system function sysctlbyname directly, rather than the intermediary sysctl terminal utility. Obtaining the free memory is done with the host_statistics64 function, which is what vm_stat uses underneath the hood as well. However, what it does with those values is different than Neofetch: the crate obtains the system's internal count of free pages, and then (saturatedly) subtracts speculative pages from them.

Speculative pages hold data grabbed ahead of time that the kernel thinks currently running processes might need later. But as the comment in the code says, they aren't used by anyone yet, and they've also pretty easily evicted if the memory is needed for more important uses, so they aren't really "used" in a way that causes problems. In other words, the Rust sysinfo crate's quite a bit stricter in its definition of what counts as "free" memory: any page that is occupied so far, even if it's not actually used by a program yet, is considered "used". The line between free and used memory is blurrier and more subjective than it may initially appear. So let's clear few things up.

The issue is obvious: different sources calculate the memory different ways. Neofetch adds wired, active, and compressed memory. The Rust sysinfo crate gives everything that is not strictly free. And Activity Monitor, although its' method isn't publicly available, seems to add application memory, wired memory, and compressed memory. As for which one is the most accurate, it depends on what you mean really. Perhaps sysinfo is the most strictly accurate, but it's very unhelpful for day-to-day usage.