Why are the output addresses and! Heap don't match?

Question

I am debugging the following C program in Windbg:

int main() { size_t size = 500*1024*1024; void *p = malloc(size); memset(p, 'a', size); printf("%p", p); } 

I compiled the program using: cl / Zi leak.c and the leak.exe file is created.

I set a breakpoint on the printf line. I run the following command:

 0:000> !address -summary --- Usage Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotal Free 21 5f0f6000 ( 1.485 Gb) 74.27% Heap 3 1f501000 ( 501.004 Mb) 95.07% 24.46% <unknown> 39 1436000 ( 20.211 Mb) 3.84% 0.99% Image 35 300000 ( 3.000 Mb) 0.57% 0.15% MappedFile 4 182000 ( 1.508 Mb) 0.29% 0.07% Stack 3 100000 ( 1.000 Mb) 0.19% 0.05% Other 6 3f000 ( 252.000 kb) 0.05% 0.01% TEB 1 1000 ( 4.000 kb) 0.00% 0.00% PEB 1 1000 ( 4.000 kb) 0.00% 0.00% --- Type Summary (for busy) ------ RgnCount ----------- Total Size -------- %ofBusy %ofTotal MEM_PRIVATE 21 207f0000 ( 519.938 Mb) 98.66% 25.39% MEM_IMAGE 64 54c000 ( 5.297 Mb) 1.01% 0.26% MEM_MAPPED 7 1be000 ( 1.742 Mb) 0.33% 0.09% --- State Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotal MEM_FREE 21 5f0f6000 ( 1.485 Gb) 74.27% MEM_COMMIT 73 1f9c9000 ( 505.785 Mb) 95.98% 24.70% MEM_RESERVE 19 1531000 ( 21.191 Mb) 4.02% 1.03% --- Protect Summary (for commit) - RgnCount ----------- Total Size -------- %ofBusy %ofTotal PAGE_READWRITE 27 1f45f000 ( 500.371 Mb) 94.95% 24.43% PAGE_EXECUTE_READ 9 376000 ( 3.461 Mb) 0.66% 0.17% PAGE_READONLY 26 1e4000 ( 1.891 Mb) 0.36% 0.09% PAGE_WRITECOPY 9 c000 ( 48.000 kb) 0.01% 0.00% PAGE_READWRITE|PAGE_GUARD 2 4000 ( 16.000 kb) 0.00% 0.00% --- Largest Region by Usage ----------- Base Address -------- Region Size ---------- Free 1fb51000 5590f000 ( 1.337 Gb) Heap 750000 1f401000 ( 500.004 Mb) <unknown> 7f0e0000 f00000 ( 15.000 Mb) Image 77bc0000 d6000 ( 856.000 kb) MappedFile 7efe5000 fb000 (1004.000 kb) Stack 210000 fd000 (1012.000 kb) Other 7efa0000 33000 ( 204.000 kb) TEB 7efdd000 1000 ( 4.000 kb) PEB 7efde000 1000 ( 4.000 kb) 

And I see that a bunch of about 500 MB, this is as expected.

But the team! heap cannot see this information:

There is only 1 bunch.

 0:000> !heap Index Address Name Debugging options enabled 1: 00650000 tail checking free checking validate parameters 0:000> !heap -a 00650000 Index Address Name Debugging options enabled 1: 00650000 Segment at 00650000 to 00750000 (0000f000 bytes committed) // Why so few memory committed. Flags: 40000062 ForceFlags: 40000060 Granularity: 8 bytes Segment Reserve: 00100000 Segment Commit: 00002000 DeCommit Block Thres: 00000200 DeCommit Total Thres: 00002000 Total Free Size: 00000517 Max. Allocation Size: 7ffdefff Lock Variable at: 00650138 Next TagIndex: 0000 Maximum TagIndex: 0000 Tag Entries: 00000000 PsuedoTag Entries: 00000000 Virtual Alloc List: 006500a0 Unable to read nt!_HEAP_VIRTUAL_ALLOC_ENTRY structure at 00750000 Uncommitted ranges: 00650090 0065f000: 000f1000 (987136 bytes) FreeList[ 00 ] at 006500c4: 0065b340 . 0065b340 0065b338: 00458 . 028b8 [104] - free Segment00 at 00650000: Flags: 00000000 Base: 00650000 First Entry: 00650588 Last Entry: 00750000 Total Pages: 00000100 Total UnCommit: 000000f1 Largest UnCommit:00000000 UnCommitted Ranges: (1) 0:000> dt p Local var @ 0x30ff04 Type void* 0x00750020 Void 0:000> !heap -p -a 0x00750020 address 00750020 found in _HEAP @ 650000 HEAP_ENTRY Size Prev Flags UserPtr UserSize - state 00750018 3e80200 0000 [00] 00750020 1f400000 - (busy VirtualAlloc) 0:000> !heap -s NtGlobalFlag enables following debugging aids for new heaps: tail checking free checking validate parameters LFH Key : 0x343f9ad2 Termination on corruption : ENABLED Heap Flags Reserv Commit Virt Free List UCR Virt Lock Fast (k) (k) (k) (k) length blocks cont. heap ----------------------------------------------------------------------------- Virtual block: 00fa0000 - 00fa0000 (size 00000000) 006d0000 40000062 1024 36 1024 1 1 1 1 0 ----------------------------------------------------------------------------- 

Why can't I see the information above in !heap -s ? How to dump all records in a heap?

Answer 1

First you need a little explanation ...

Heap vs virtual memory

The IMHO heap was invented because the graininess of VirtualAlloc() (which is 64 KB) was too wasteful for small amounts of data. If I look at the output !heap <address> , I see

 Heap entries for Segment00 in Heap 00100000 address: psize . size flags state (requested size) 00100000: 00000 . 00588 [101] - busy (587) 

See the size column, which contains 5 digits. This probably means the maximum usable heap record size

 0:001> ? fffff Evaluate expression: 1048575 = 000fffff 

approximately 1 MB. Above this size, it probably doesn't make sense to have heap grain, and you can use VirtualAlloc() directly. Or, not you, but the malloc() function solves this for you.

Exit error message

In your release you noticed two lines

 Virtual Alloc List: 006500a0 Unable to read nt!_HEAP_VIRTUAL_ALLOC_ENTRY structure at 00750000 

This is an indicator that there are parts in the heap that are actually being processed by VirtualAlloc() . However, WinDbg cannot find a data type for it:

 0:001> dt nt!_HEAP_VIRTUAL_ALLOC_ENTRY Symbol nt!_HEAP_VIRTUAL_ALLOC_ENTRY not found. 

There is no output !heap -s , so I created it on my machine (WinDbg 6.3.9600.16384):

 0:001> !heap 00100000 -s LFH Key : 0x62502d13 Termination on corruption : ENABLED Heap Flags Reserv Commit Virt Free List UCR Virt Lock Fast (k) (k) (k) (k) length blocks cont. heap ----------------------------------------------------------------------------- Virtual block: 005b0000 - 005b0000 (size 00000000) Virtual block: 006b0000 - 006b0000 (size 00000000) ... Virtual block: 1fab0000 - 1fab0000 (size 00000000) 00100000 00000002 1024 204 1024 4 7 1 500 0 LFH ----------------------------------------------------------------------------- 

In my demo, I created ~ 500 blocks of 1 MB each. Please note that size 00000000 seems to be broken. However, if I use !address in the block, it will get the right size of the region (00100000):

 0:001> !address 005b0000 ... Usage: Heap Base Address: 005b0000 End Address: 006b0000 Region Size: 00100000 State: 00001000 MEM_COMMIT Protect: 00000004 PAGE_READWRITE Type: 00020000 MEM_PRIVATE Allocation Base: 005b0000 Allocation Protect: 00000004 PAGE_READWRITE 

What might the missing _HEAP_VIRTUAL_ALLOC_ENTRY look like?

 0:001> dd 770000 L10 00770000 00870000 00670000 00000000 00000000 00770010 00100000 00100000 c8d50110 04000000 00770020 003df5a8 00870020 00000000 00000000 00770030 000ffc00 00000001 000000c1 fdfdfdfd 

A value of 870000 at offset 0 is similar to FLink on the next block of memory and a value of 670000 at offset 4 a BLink on the previous one.

Offsets 10 and 14 match the size of the area.

 0:001> ? 0n1023*0n1024 Evaluate expression: 1047552 = 000ffc00 

At an offset of +30 - the size (here I highlighted blocks of 1023 kB).

FDFDFDFD is a well-known debugging magic number that indicates the absence of ground, so this is most likely the end of the structure.

Extract structure from XP dump

I have a dump available in Windows XP. See how nt!_HEAP_VIRTUAL_ALLOC_ENTRY :

 0: kd> dt -r1 nt!_HEAP_VIRTUAL_ALLOC_ENTRY +0x000 Entry : _LIST_ENTRY +0x000 Flink : Ptr32 _LIST_ENTRY +0x004 Blink : Ptr32 _LIST_ENTRY +0x008 ExtraStuff : _HEAP_ENTRY_EXTRA +0x000 AllocatorBackTraceIndex : Uint2B +0x002 TagIndex : Uint2B +0x004 Settable : Uint4B +0x000 ZeroInit : Uint8B +0x010 CommitSize : Uint4B +0x014 ReserveSize : Uint4B +0x018 BusyBlock : _HEAP_ENTRY +0x000 Size : Uint2B +0x002 PreviousSize : Uint2B +0x000 SubSegmentCode : Ptr32 Void +0x004 SmallTagIndex : UChar +0x005 Flags : UChar +0x006 UnusedBytes : UChar +0x007 SegmentIndex : UChar 

Conclusion

!heap seems to have problems for large blocks created by malloc() , which actually use VirtualAlloc() in this case. WinDbg cannot find the data type that it expects to match memory contents. Microsoft should probably fix this error.

The output statistics for the heap !address -summary seems reasonable.