Information about PTE (Page Table Entries) in Windows
For more convenient buffer overflows, I change our dedicated memory allocator to allocate a full 4KB page, not just the number of bytes needed. Then I change the protection and page size so that if the caller writes before or after the allocated portion of memory, the application crashes immediately.
The problem is, even though I have enough memory, the application never starts completely due to insufficient memory. This has two reasons:
- since each allocation requires 4KB, we'll probably hit the 2GB limit very soon. This problem can be solved if I make a 64 bit executable (haven't tried it yet).
- even when I only need a few hundred megabytes, the allocation is interrupted at some point.
The second problem is the biggest one and I think it has to do with the maximum number of PTEs (entries in the page table that store information about how virtual memory maps to physical memory and whether pages should be read-only or not) you may have in the process.
My questions (or hints):
- Where can I find information on the maximum number of PTEs in a process?
- Is this different (higher) for 64-bit systems / applications or not?
- Can the PTE number be configured in the application or in Windows?
Thanks,
Patrick
PS. note to those who try to argue that you shouldn't write your own memory manager:
- My application is quite specific, so I really want to have complete control over memory management (can't give more details)
- We had a memory overwrite last week that we couldn't find using the standard C ++ manager and C / C ++ runtime debugging functionality (it only said "lock corrupted" minutes after actual corruption ")
- We also tried standard Windows utilities (like GFLAGS, ...), but they slowed down the application by a factor of 100 and couldn't find the exact rewrite position, either
- We also tried the "Full Page" functionality in Application Verifier, but then the application also fails to launch (may not work from PTE either).
a source to share
The shotgun approach is to distribute these isolated 4K bytes in random order. This means that you will need to repeat the same tests with the same input. Sometimes it will catch the error if you're lucky.
A slightly smarter approach is to use a different algorithm than just random - for example. make it dependent on the call stack whether an allocation is allocated. For example, do you trust std::string
users, and use suspicious raw ones malloc
?
a source to share
There is what I thought was an excellent series of blog posts by Mark Russinovich on a technique called "Pushing limits of Windows ..."
http://blogs.technet.com/markrussinovich/archive/2008/07/21/3092070.aspx
It has several articles on virtual memory, paged unpaid memory, physical memory, and others.
He mentions small utilities that he uses to measure system resources.
I hope you find answers to them.
a source to share
To find a simpler buffer overflow, I'm changing our memory custom to allocate a full 4KB page, not just the number of bytes required.
This has already been done. Application Verifier with PageHeap .
Information on PTE and memory architecture can be found in Windows Internals, 5th Ed. and Intel Manuals .
Is this different (higher) for 64 bit systems / applications or not?
Sure. 64-bit Windows has a much larger address space, so more PTEs are required to display it.
Where can I find information on the maximum number of PTEs in the process?
This is not as important as the maximum amount of user address space available in the process. (The PTE number is the number divided by the page size.)
That's 2 GB on 32-bit Windows and much more on x64 Windows. (The actual number varies, but it is "large enough").
The problem is that although I have enough memory, the application never starts completely because it runs out of memory.
Are you a) memory leak? b) using horribly inefficient algorithms?
a source to share
Take a look at the OpenBSD malloc implementation . Many of the same ideas (and more) implemented by very experienced people.
a source to share