Companion pages / physical memory in Java
My goal is to provide array allocation in java in contiguous physical memory. The problem I am facing is that the pages the array is allocated to tend not to be contiguous in physical memory unless I allocate a really large array.
My questions:
- Why does a really large array provide pages that are contiguous in physical memory?
- Is there a way to ensure that an array is allocated in physical memory that doesn't require a large array?
- How can I determine which page or physical address exists in a Java object / array without measuring cache / cache misses?
I am not looking for answers asking why I am doing this in java. I understand that C will "solve my problem" and that I am going against the fundamental nature of Java. Nevertheless, I have every reason for this.
Answers don't have to be guaranteed to work all the time. I'm looking for answers that work most of the time. Extra points for creative, ready-made answers that no reasonable Java programmer would ever write. This is normal for the platform (x86 32-bit 64-bit).
a source to share
Considering that the garbage collector moves objects around in (logical) memory, I guess you will be out of luck.
About what you can do is use ByteBuffer.allocateDirect . This will (usually) not move across the logical memory of the GC, but can be moved into physical memory or even paged out to disk. If you want better guarantees, you'll have to click on OS.
Having said that, if you can set the page size to be larger than your heap, then all arrays are bound to be physically contiguous (or replaced).
a source to share
No. Physically contiguous memory requires direct interaction with the OS. Most JVM applications are included only to obtain practically contiguous addresses. And the JVM can't give you what it doesn't get from the OS.
Also, why would you need it? If you are setting up DMA transfers, you are probably using methods other than Java anyway.
Background bit:
Physical memory on a modern PC is usually flexible, on replaceable DIMMs. Each byte has a physical address, so the operating system determines at boot time which physical addresses are available. It turned out that applications are better off not using these addresses directly. Instead, all modern processors (and their caches) use virtual addresses. There is a mapping table for physical addresses, but this is not required to complete. Exchange to disk is permitted using virtual addresses that are not tied to physical addresses. Another level of flexibility is achieved by having one table for each process with incomplete mappings. If process A has a virtual address that maps to physical address X, but process B is down, then there is no way that process B can write to physical address X.and we can consider this memory exclusive to processing A. Obviously for this to be safe, the OS must protect access to the mapping table, but all modern OSs do.
The mapping table works at the page level. A page or contiguous subset of physical addresses is mapped to a contiguous subset of virtual addresses. The trade-off between overhead and granularity resulted in 4K pages being the overall page size. But since each page has its own mapping, you cannot allow adjacency beyond that page size. In particular, when pages are unloaded from physical memory, swapped out to disk and rebuilt, it is possible that they end up at a new physical memory address. The program does not notice because the virtual address does not change, only the OS-managed mapping table is executed.
a source to share
There may be ways to trick a particular JVM into what you want, but this is likely to be fragile, complex, and most likely very specific to the JVM, its version, the OS it runs on, etc. In other words, wasted effort.
Therefore, without knowing more about your problem, I don't think anyone can help. Of course, there is no way to do this in Java in general, at best on a specific JVM.
To suggest an alternative:
If you really need to store data in contiguous memory, why not do it in a small C library and call it through JNI?
a source to share
As I can see. You have not yet explained why
- that primitive arrays are not contiguous in memory. I don't understand why they won't be contiguous in virtual memory. (cf Arrays of object are unlikely to have contiguous objects in memory)
- an array that is not contiguous in physical memory (RAM, that is, RAM) will make a significant difference in performance. for example a measurable difference in the performance of your application.
What it seems you are really looking for a low-level way to allocate arrays, because you are used to doing it in C and performance is a requirement for that.
BTW: Accessing ByteBuffer.allocateDirect () with getDouble () / putDouble () can be slower than just using double [], since the former involves JNI calls and the latter can be optimized without calling at all.
The reason it is used is for data exchange between Java and C spaces for example NIO calls. It only works well when read / write is minimized. Otherwise, you are better off using something in the Java space.
i.e. If you don’t understand what you’re doing and why you’re doing it, you may end up with a solution that might make you feel better, but is actually harder and worse than a simple solution.
a source to share
Please note this answer to a related question, which discusses System.identityHashCode () and identifying an object's memory address. The bottom line is that you can use the default hashCode () implementation to determine the source memory address of the array (assuming it is set to int / 32-bit)
a source to share