Unmanaged memory leaks in managed applications

crazy newbie

I am trying to find the root cause of an unmanaged heap. My application is a .NET application, but it calls some 3rd party libraries for C++. I took two memory snapshots 1.5GB apart.

This is the first snapshot! address -summary output

0:000> !address -summary

--- Usage Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotal
Free                                   1144      7fa`6695c000 (   7.978 Tb)           99.73%
<unknown>                              1498        4`4ba12000 (  17.182 Gb)  76.71%    0.21%
Heap                                   1507        1`13c3c000 (   4.309 Gb)  19.24%    0.05%
Stack                                  2385        0`31540000 ( 789.250 Mb)   3.44%    0.01%
Image                                  1289        0`0831a000 ( 131.102 Mb)   0.57%    0.00%
TEB                                     794        0`00634000 (   6.203 Mb)   0.03%    0.00%
Other                                    14        0`001b7000 (   1.715 Mb)   0.01%    0.00%
PEB                                       1        0`00001000 (   4.000 kb)   0.00%    0.00%

--- Type Summary (for busy) ------ RgnCount ----------- Total Size -------- %ofBusy %ofTotal
MEM_PRIVATE                            5613        5`8adfb000 (  22.170 Gb)  98.99%    0.27%
MEM_IMAGE                              1822        0`0bbfa000 ( 187.977 Mb)   0.82%    0.00%
MEM_MAPPED                               53        0`02c9f000 (  44.621 Mb)   0.19%    0.00%

--- State Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotal
MEM_FREE                               1144      7fa`6695c000 (   7.978 Tb)           99.73%
MEM_RESERVE                            2121        4`07c5c000 (  16.121 Gb)  71.98%    0.20%
MEM_COMMIT                             5367        1`91a38000 (   6.276 Gb)  28.02%    0.08%

--- Protect Summary (for commit) - RgnCount ----------- Total Size -------- %ofBusy %ofTotal
PAGE_READWRITE                         3216        1`831df000 (   6.049 Gb)  27.01%    0.07%
PAGE_EXECUTE_READ                       283        0`095c5000 ( 149.770 Mb)   0.65%    0.00%
PAGE_READONLY                           846        0`03242000 (  50.258 Mb)   0.22%    0.00%
PAGE_EXECUTE_READWRITE                  143        0`00f9b000 (  15.605 Mb)   0.07%    0.00%
PAGE_READWRITE|PAGE_GUARD               795        0`00f2b000 (  15.168 Mb)   0.07%    0.00%
PAGE_WRITECOPY                           83        0`00188000 (   1.531 Mb)   0.01%    0.00%
PAGE_EXECUTE                              1        0`00004000 (  16.000 kb)   0.00%    0.00%

--- Largest Region by Usage ----------- Base Address -------- Region Size ----------
Free                                      5`c0090000      7f8`da150000 (   7.972 Tb)
<unknown>                                 3`9c0ee000        0`e3fa2000 (   3.562 Gb)
Heap                                      0`09a90000        0`00fd0000 (  15.813 Mb)
Stack                                     0`00cf0000        0`000fc000 (1008.000 kb)
Image                                   7fe`fe0aa000        0`0089e000 (   8.617 Mb)
TEB                                     7ff`ff7bc000        0`00002000 (   8.000 kb)
Other                                     0`007f0000        0`00181000 (   1.504 Mb)
PEB                                     7ff`fffd3000        0`00001000 (   4.000 kb)

This is the second snapshot! address -summary output

0:000> !address -summary

--- Usage Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotal
Free                                   1129      7fa`2cad7000 (   7.977 Tb)           99.72%
<unknown>                              1520        4`4ca9a000 (  17.198 Gb)  73.80%    0.21%
Heap                                   1585        1`486b3000 (   5.132 Gb)  22.02%    0.06%
Stack                                  2586        0`35840000 ( 856.250 Mb)   3.59%    0.01%
Image                                  1275        0`0831a000 ( 131.102 Mb)   0.55%    0.00%
TEB                                     861        0`006ba000 (   6.727 Mb)   0.03%    0.00%
Other                                    14        0`001b7000 (   1.715 Mb)   0.01%    0.00%
PEB                                       1        0`00001000 (   4.000 kb)   0.00%    0.00%

--- Type Summary (for busy) ------ RgnCount ----------- Total Size -------- %ofBusy %ofTotal
MEM_PRIVATE                            5978        5`c3c7d000 (  23.059 Gb)  98.96%    0.28%
MEM_IMAGE                              1810        0`0bbfa000 ( 187.977 Mb)   0.79%    0.00%
MEM_MAPPED                               54        0`03ca2000 (  60.633 Mb)   0.25%    0.00%

--- State Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotal
MEM_FREE                               1129      7fa`2cad7000 (   7.977 Tb)           99.72%
MEM_RESERVE                            2218        3`ea250000 (  15.659 Gb)  67.20%    0.19%
MEM_COMMIT                             5624        1`e92c9000 (   7.643 Gb)  32.80%    0.09%

--- Protect Summary (for commit) - RgnCount ----------- Total Size -------- %ofBusy %ofTotal
PAGE_READWRITE                         3419        1`da924000 (   7.415 Gb)  31.82%    0.09%
PAGE_EXECUTE_READ                       283        0`095c5000 ( 149.770 Mb)   0.63%    0.00%
PAGE_READONLY                           846        0`03242000 (  50.258 Mb)   0.21%    0.00%
PAGE_READWRITE|PAGE_GUARD               862        0`01071000 (  16.441 Mb)   0.07%    0.00%
PAGE_EXECUTE_READWRITE                  148        0`00fec000 (  15.922 Mb)   0.07%    0.00%
PAGE_WRITECOPY                           65        0`0013d000 (   1.238 Mb)   0.01%    0.00%
PAGE_EXECUTE                              1        0`00004000 (  16.000 kb)   0.00%    0.00%

--- Largest Region by Usage ----------- Base Address -------- Region Size ----------
Free                                      5`e2b3f000      7f8`b76a1000 (   7.972 Tb)
<unknown>                                 1`a5991000        0`da6ff000 (   3.413 Gb)
Heap                                      0`09a90000        0`00fd0000 (  15.813 Mb)
Stack                                     0`00cf0000        0`000fc000 (1008.000 kb)
Image                                   7fe`fe0aa000        0`0089e000 (   8.617 Mb)
TEB                                     7ff`ff7bc000        0`00002000 (   8.000 kb)
Other                                     0`007f0000        0`00181000 (   1.504 Mb)
PEB                                     7ff`fffd3000        0`00001000 (   4.000 kb)

Based on these, it looks like the native heap has grown by 1GB. So let's look at the native heap. This is the first snapshot

0:000> !heap -s
LFH Key                   : 0x0000007b0b4a5afd
Termination on corruption : ENABLED
          Heap     Flags   Reserv  Commit  Virt   Free  List   UCR  Virt  Lock  Fast 
                            (k)     (k)    (k)     (k) length      blocks cont. heap 
-------------------------------------------------------------------------------------
Virtual block: 00000000031d0000 - 00000000031d0000 (size 0000000000000000)
Virtual block: 0000000003690000 - 0000000003690000 (size 0000000000000000)
Virtual block: 0000000024bf0000 - 0000000024bf0000 (size 0000000000000000)
Virtual block: 00000001412c0000 - 00000001412c0000 (size 0000000000000000)
Virtual block: 0000000158120000 - 0000000158120000 (size 0000000000000000)
Virtual block: 0000000102bd0000 - 0000000102bd0000 (size 0000000000000000)
Virtual block: 0000000021a50000 - 0000000021a50000 (size 0000000000000000)
Virtual block: 000000011b230000 - 000000011b230000 (size 0000000000000000)
Virtual block: 0000000073f40000 - 0000000073f40000 (size 0000000000000000)
0000000000090000 00000002  695936 693228 695936   2304  2251    47    9     e6   LFH
0000000000010000 00008000      64      4     64      1     1     1    0      0      
0000000000410000 00001002    1088    952   1088      7     4     2    0      0   LFH
0000000000550000 00041002     512      8    512      3     1     1    0      0      
0000000000020000 00001002    1088   1044   1088     10     5     2    0      0   LFH
00000000005e0000 00041002     512    256    512      2     2     1    0      0   LFH
0000000000b40000 00001002    1536    680   1536      4    22     2    0      0   LFH
0000000000c70000 00041002     512      8    512      3     1     1    0      0      
0000000000a80000 00001002     512      8    512      3     1     1    0      0      
0000000000530000 00001002      64     24     64      8     2     1    0      0      
0000000001100000 00001002  355456 190148 355456  96748   658    89    0      0   LFH
    External fragmentation  50 % (658 free blocks)
0000000001550000 00001002   15424   7084  15424   2921   103     7    0      2   LFH
    External fragmentation  41 % (103 free blocks)
0000000005c30000 00001002    1536    608   1536      6    31     2    0      0   LFH
0000000003d00000 00001002    1536    592   1536      4     9     2    0      0   LFH
0000000003df0000 00001002      64      8     64      3     1     1    0      0      
0000000003ff0000 00001002      64      8     64      3     1     1    0      0      
0000000005e90000 00011002     512    108    512     20    12     1    0      1      
0000000005f20000 00001002     512      8    512      3     2     1    0      0      
0000000008bc0000 00001002  695936 685784 695936    410   701    47    0      5   LFH
Virtual block: 00000000116e0000 - 00000000116e0000 (size 0000000000000000)
Virtual block: 000000001c7c0000 - 000000001c7c0000 (size 0000000000000000)
0000000006580000 00001002 2703360 2700036 2703360 122093  8251   176    2  1222d   LFH
0000000014730000 00001002    1088    328   1088     67     7     2    0      0   LFH
000000004fe50000 00001002     512      8    512      3     1     1    0      0      
000000004fdb0000 00001002     512      8    512      3     1     1    0      0      
0000000010170000 00001002     512     12    512      1     2     1    0      0      
0000000058960000 00001002     512      8    512      1     3     1    0      0      
00000000104e0000 00001002     512    264    512      3     5     1    0      0   LFH
-------------------------------------------------------------------------------------

This is the second snapshot

0:000> !heap -s
LFH Key                   : 0x0000007b0b4a5afd
Termination on corruption : ENABLED
          Heap     Flags   Reserv  Commit  Virt   Free  List   UCR  Virt  Lock  Fast 
                            (k)     (k)    (k)     (k) length      blocks cont. heap 
-------------------------------------------------------------------------------------
Virtual block: 00000000031d0000 - 00000000031d0000 (size 0000000000000000)
Virtual block: 0000000003690000 - 0000000003690000 (size 0000000000000000)
Virtual block: 0000000024bf0000 - 0000000024bf0000 (size 0000000000000000)
Virtual block: 0000000158120000 - 0000000158120000 (size 0000000000000000)
Virtual block: 000000011b230000 - 000000011b230000 (size 0000000000000000)
Virtual block: 000000002d9c0000 - 000000002d9c0000 (size 0000000000000000)
Virtual block: 0000000082f60000 - 0000000082f60000 (size 0000000000000000)
Virtual block: 00000000273a0000 - 00000000273a0000 (size 0000000000000000)
Virtual block: 0000000073e90000 - 0000000073e90000 (size 0000000000000000)
0000000000090000 00000002  857856 847980 857856   3491  2443    57    9    145   LFH
0000000000010000 00008000      64      4     64      1     1     1    0      0      
0000000000410000 00001002    1088    952   1088      7     4     2    0      0   LFH
0000000000550000 00041002     512      8    512      3     1     1    0      0      
0000000000020000 00001002    1088   1044   1088     10     5     2    0      0   LFH
00000000005e0000 00041002     512    256    512      2     2     1    0      0   LFH
0000000000b40000 00001002    1536    780   1536      7    26     2    0      0   LFH
0000000000c70000 00041002     512      8    512      3     1     1    0      0      
0000000000a80000 00001002     512      8    512      3     1     1    0      0      
0000000000530000 00001002      64     24     64      8     2     1    0      0      
0000000001100000 00001002  355456 190148 355456  96735   658    89    0      0   LFH
    External fragmentation  50 % (658 free blocks)
0000000001550000 00001002   15424   7084  15424   2915    97     7    0      2   LFH
    External fragmentation  41 % (97 free blocks)
0000000005c30000 00001002    1536    616   1536      9    33     2    0      0   LFH
0000000003d00000 00001002    1536    592   1536      4     9     2    0      0   LFH
0000000003df0000 00001002      64      8     64      3     1     1    0      0      
0000000003ff0000 00001002      64      8     64      3     1     1    0      0      
0000000005e90000 00011002     512    108    512     20    13     1    0      9      
0000000005f20000 00001002     512      8    512      3     2     1    0      0      
0000000008bc0000 00001002  857856 852992 857856    426   745    57    0      5   LFH
Virtual block: 00000000116e0000 - 00000000116e0000 (size 0000000000000000)
Virtual block: 00000000124b0000 - 00000000124b0000 (size 0000000000000000)
Virtual block: 000000011bd10000 - 000000011bd10000 (size 0000000000000000)
0000000006580000 00001002 3237696 3221472 3237696 150399  9556   209    3  1b6e4   LFH
0000000014730000 00001002    1088    328   1088     67     7     2    0      0   LFH
000000004fe50000 00001002     512      8    512      3     1     1    0      0      
000000004fdb0000 00001002     512      8    512      3     1     1    0      0      
0000000010170000 00001002     512     12    512      1     2     1    0      0      
0000000058960000 00001002     512      8    512      1     3     1    0      0      
0000000010430000 00001002     512    264    512      3     5     1    0      0   LFH
-------------------------------------------------------------------------------------

If I read this data correctly, the memory size of the heap @0000000006580000 increases the most. Let's take a closer look at this heap

Take the first snapshot again

0:000> !heap -stat -h 0000000006580000 
 heap @ 0000000006580000
group-by: TOTSIZE max-display: 20
    size     #blocks     total     ( %) (percent of total busy bytes)
    2c48 1 - 2c48  (26.79)
    280 c - 1e00  (18.15)
    c0 22 - 1980  (15.43)
    38 40 - e00  (8.47)
    30 40 - c00  (7.26)
    40 19 - 640  (3.78)
    18 32 - 4b0  (2.84)
    20 25 - 4a0  (2.80)
    428 1 - 428  (2.51)
    12 2d - 32a  (1.91)
    328 1 - 328  (1.91)
    8 58 - 2c0  (1.66)
    c 2d - 21c  (1.28)
    d8 2 - 1b0  (1.02)
    1c a - 118  (0.66)
    3e 4 - f8  (0.59)
    3c 4 - f0  (0.57)
    d0 1 - d0  (0.49)
    bc 1 - bc  (0.44)
    b8 1 - b8  (0.43)

Here's how to find the second snapshot

heap @ 0000000006580000
group-by: TOTSIZE max-display: 20
    size     #blocks     total     ( %) (percent of total busy bytes)
    2c48 1 - 2c48  (26.80)
    280 c - 1e00  (18.16)
    c0 22 - 1980  (15.44)
    38 40 - e00  (8.47)
    30 40 - c00  (7.26)
    40 19 - 640  (3.78)
    18 32 - 4b0  (2.84)
    20 25 - 4a0  (2.80)
    428 1 - 428  (2.52)
    328 1 - 328  (1.91)
    12 2c - 318  (1.87)
    8 58 - 2c0  (1.66)
    c 2d - 21c  (1.28)
    d8 2 - 1b0  (1.02)
    1c a - 118  (0.66)
    3e 4 - f8  (0.59)
    3c 4 - f0  (0.57)
    d0 1 - d0  (0.49)
    bc 1 - bc  (0.44)
    b8 1 - b8  (0.44)

Is this the correct way to find the cause of the memory leak, as I don't see a matching % change in busy bytes explaining the ~0.5GB memory increase for this particular heap? Also, how do I determine what objects/modules are in that heap, and who is doing the allocations?

Josh Polly

Since you can take snapshots, I recommend using UMDH , which is part of the debugger toolset that comes with WinDbg. There are many tutorials/examples out there, but the basic flow is as follows:

  1. Enable stack trace logging:

    gflags.exe -i processName.exe +ust

  2. Run the process, and note the "Process ID" (in the task manager)

  3. Take the first memory snapshot (replace 1234 with the process ID)

    umdh.exe -p:1234 -f:mem1.txt

  4. Let the process run for a while (and leak memory)

  5. Take a second snapshot

    umdh.exe -p:1234 -f:mem2.txt

  6. Create a report with size and stack trace

    umdh.exe mem1.txt mem2.txt >mem_compare.txt

  7. Disable the stack trace of the procedure

    gflags.exe -i processName.exe -ust

The report will look like this:

+ a556ae5 ( a556ae5 -     0)      7 allocs    BackTraceC0E
+       7 (     7 -     0)    BackTraceC0E    allocations
 
    ntdll!RtlLogStackBackTrace+00000007
    ntdll!RtlAllocateHeap+0000023A
    ntdll!RtlDebugAllocateHeap+000000B5
    ntdll!RtlpAllocateHeap+000000C4
    ntdll!RtlAllocateHeap+0000023A
    ntdll!LdrpTagAllocateHeap+00000025
    ntdll!LdrpTagAllocateHeap29+00000015
    processName!malloc+00000016 
    processName!createLotsOfObjects+00000712
    ...

This shows that between the two snapshots, about 170megs (0xa55ae5 bytes) of memory was allocated with this stack trace.

Related

Unmanaged memory leaks in managed applications

crazy newbie I am trying to find the root cause of an unmanaged heap. My application is a .NET application, but it calls some 3rd party libraries for C++. I took two memory snapshots 1.5GB apart. This is the first snapshot! address -summary output 0:000> !addr

Store managed references in unmanaged memory

puppy Is it possible to store references to regular managed objects (not pinned) in unmanaged storage? Essentially, I don't want to pin the object, but mark the zone as GC root or something like that. I've looked at the GCHandle documentation, but it's not cle

Store managed references in unmanaged memory

puppy Is it possible to store references to regular managed objects (not pinned) in unmanaged storage? Essentially, I don't want to pin the object, but mark the zone as GC root or something like that. I've looked at the GCHandle documentation, but it's not cle

Store managed references in unmanaged memory

puppy Is it possible to store references to regular managed objects (not pinned) in unmanaged storage? Essentially, I want to mark the region as a GC root or similar, not a pinned object. I've looked at the GCHandle documentation, but it's not clear if it actu

Store managed references in unmanaged memory

puppy Is it possible to store references to regular managed objects (not pinned) in unmanaged storage? Essentially, I want to mark the region as a GC root or similar, not a pinned object. I've looked at the GCHandle documentation, but it's not clear if it actu

Clean up memory leaks in unmanaged code

Moez Rebai I've been wandering about the best way to resolve sandboxed memory leaks in 3rd party libraries when I'm using them from a .Net framework. I have a 3rd party library written in C++/cli that leaks a lot of memory. Has anyone found a good way to clean

Clean up memory leaks in unmanaged code

Moez Rebai I've been wandering about the best way to resolve sandboxed memory leaks in 3rd party libraries when I'm using them from a .Net framework. I have a 3rd party library written in C++/cli that leaks a lot of memory. Has anyone found a good way to clean

Clean up memory leaks in unmanaged code

Moez Rebai I've been wandering about the best way to resolve sandboxed memory leaks in 3rd party libraries when I'm using them from a .Net framework. I have a 3rd party library written in C++/cli that leaks a lot of memory. Has anyone found a good way to clean

Unmanaged memory profiling of UWP applications?

Skynet 094 I've been looking for a tool or technique, or any technique that can help me find which objects are being allocated in unmanaged heap memory from a UWP app. I tried looking at WPR/WPA combo to track objects. However, nothing useful was found there.

Unmanaged memory profiling of UWP applications?

Skynet 094 I've been looking for a tool or technique, or any technique that can help me find what objects are being allocated in unmanaged heap memory from a UWP app. I tried looking at WPR/WPA combo to track objects. However, nothing useful was found there. (

Unmanaged memory profiling of UWP applications?

Skynet 094 I've been looking for a tool or technique, or any technique that can help me find what objects are being allocated in unmanaged heap memory from a UWP app. I tried looking at WPR/WPA combo to track objects. However, nothing useful was found there. (

Unmanaged memory profiling of UWP applications?

Skynet 094 I've been looking for a tool or technique, or any technique that can help me find what objects are being allocated in unmanaged heap memory from a UWP app. I tried looking at WPR/WPA combo to track objects. However, nothing useful was found there. (

Unmanaged memory profiling of UWP applications?

Skynet 094 I've been looking for a tool or technique, or any technique that can help me find what objects are being allocated in unmanaged heap memory from a UWP app. I tried looking at WPR/WPA combo to track objects. However, nothing useful was found there. (

Unmanaged memory profiling of UWP applications?

Skynet 094 I've been looking for a tool or technique, or any technique that can help me find what objects are being allocated in unmanaged heap memory from a UWP app. I tried looking at WPR/WPA combo to track objects. However, nothing useful was found there. (

Understanding memory leaks in Android applications

dabbler I'm very new to Java programming, have extensive C++ experience, and am reading about how references can cause memory leaks in Android applications. This explanation confuses me. In "Lesson 2" it says: The point is, the Activity doesn't know that the l

Understanding memory leaks in Android applications

dabbler I'm very new to Java programming, have extensive C++ experience, and am reading about how references can cause memory leaks in Android applications. This explanation confuses me. In "Lesson 2" it says: The point is, the Activity doesn't know that the l

Find memory leaks in Go applications

rumor: I have a simple program that reads frames from a webcam and puts pointers to them on a channel in a goroutine. The other part reads frame pointers from the channel and pops them into another queue. Another queue then performs some additional processing

Find memory leaks in Go applications

rumor: I have a simple program that reads frames from a webcam and puts pointers to them on a channel in a goroutine. The other part reads frame pointers from the channel and pops them into another queue. Another queue then performs some additional processing

Understanding memory leaks in Android applications

dabbler I'm very new to Java programming, have extensive C++ experience, and am reading about how references can cause memory leaks in Android applications. This explanation confuses me. In "Lesson 2" it says: The point is, the Activity doesn't know that the l

Understanding memory leaks in Android applications

GVillani82 I found the following code in the article "Avoiding memory leaks" : private static Drawable sBackground; @Override protected void onCreate(Bundle state) { super.onCreate(state); TextView label = new TextView(this); label.setText("Leaks are b

Understanding memory leaks in Android applications

dabbler I'm very new to Java programming, have extensive C++ experience, and am reading about how references can cause memory leaks in Android applications. This explanation confuses me. In "Lesson 2" it says: The point is, the Activity doesn't know that the l

Find memory leaks in Go applications

rumor: I have a simple program that reads frames from a webcam and puts pointers to them on a channel in a goroutine. The other part reads frame pointers from the channel and pops them into another queue. Another queue then performs some additional processing

Find memory leaks in Go applications

rumor: I have a simple program that reads frames from a webcam and puts pointers to them on a channel in a goroutine. The other part reads frame pointers from the channel and pops them into another queue. Another queue then performs some additional processing

Understanding memory leaks in Android applications

dabbler I'm very new to Java programming, have extensive C++ experience, and am reading about how references can cause memory leaks in Android applications. This explanation confuses me. In "Lesson 2" it says: The point is, the Activity doesn't know that the l

Determine if IntPtr points to managed or unmanaged memory

Wesent Wright I'm using a wrapped C library in C# and need to convert an image from that library to a Bitmap and back, but without copying the pixel buffer. Converting to a bitmap is simple: Bitmap WrapAsBitmap(CImage image) { return new Bitmap(image.Width

Remotely identify memory leaks in applications used by customers

Song Lake: A client complained that he had a memory leak in our Java application. Although I tried my best to reproduce his environment, configuration and usage, I was unable to reproduce and therefore determined the leak. I'd like to go the other way...instea

How to reduce/eliminate memory leaks in Angular applications

Jay Shukla I am optimizing my large ad Angular App. It's great to spot the problem, as I found out Google DevTools. When I first started learning DevTools, I was very confused about memory leaks. When I move back and forth to different pages in the application

How to avoid memory leaks in Mule applications?

Attila Is there something special I have to consider in order to avoid memory leaks in Mule applications ? How can we avoid memory leaks in Mule applications? For example; do we actually have to remove the flow variable? What do developers of Mule applications

Remotely identify memory leaks in applications used by customers

Song Lake: A client complained that he had a memory leak in our Java application. Although I tried my best to reproduce his environment, configuration and usage, I was unable to reproduce and therefore determined the leak. I'd like to go the other way...instea