AWE data warehouse through collections / lists / other containers

Question

Does anyone have suggestions (product, tools, methods or others) for storing and processing user data (delphi collections, binary trees, DIContainers, etc.) that DO NOT limit the standard win32 memory address space? As a last resort, is there anything off the shelf that can do the equivalent of holding a 10-liter TList, thereby blowing out the / 3GB switch barrier and 4 GB window-per-sheet?

What we ideally need is quite transparent for the Delphi application programmer, but allows very quick access to the data stored in its structures, preferably by means of key searches. The equivalent of the delphi colletion container would be great, but its memory usage should be done through AWE. You will also need to take care of displaying and decoupling the physical space that it uses in the win32 process, using it, that is, it will be bit-transit ...

Moving data to the database is not the answer - the information must remain resident for very quick access. The databases / tables in memory that we tried to use do not use AWE, and are also slow to access. Our current Delphi data structures are great, but they put restrictions on the win32 address space.

Answer 1

I’m going to be a complete winner and I’ll tell you that I did something even more perfect than what you are describing ... at work. I was all scared. I have never seen anything like it. We combine VM, AWE, MMF and (soon) 32 <64 bit IPC into one large medium-sized data processing machine, addressing up to 64 GB of memory, while processing hundreds of data sets, tens of GB each.

But I can give you some tips: AWE view-swapping is rather slow because it forcibly pauses all current threads during the exchange. Therefore, choose the size of your window wisely (the smaller the faster the swap - but the overhead is lower with large sizes). We set AWE sizes equal to the default Windows page size (4 KB), but only because this choice is made in random order. Linear data access can be faster with larger sizes.

Each view can be displayed in any part of the allocated AWE memory, so one thing that can help is to map only these pages to the view that you need to access, and try to save the views in unprotected views (the priority queue comes to mind).

In addition, your design should have a registration engine that supports the relationship between the view and AWE memory. And it's better to be thread safe!

As for general use: No, this does not correspond to the usual Delphi classes. You must completely switch to another concept - and base your data structures on this.

Anyway, good luck! You will need it ...; -)

Answer 2

There are system calls that can do this, but are not supported in all versions of Windows (in particular, Windows XP does not support TRYPT).

Transparency would be a bit of a problem since the API could not return pointers to objects. Matching more than 4 GB of RAM to a 4 GB address space means that a 32-bit pointer can be ambiguous - you can potentially map different objects in one place.

This ambiguity means that you will have to create proxies for objects that contain a handle that you can use to access the "record". Some versions of SQL Server use this technique to store disk buffers in AWE memory. This approach is likely to work for something like rows in a matrix where operations are performed on the entire row. Finer access will be more difficult.

To provide direct access to the displayed object, you will need to implement a protocol in which a temporary pointer to the associated memory would be available. It would also require the object to be locked in memory during use - again, the punch goes to your transparency.

Assuming you can get a 64-bit version of Delphi, now you might be better off upgrading to a 64-bit version of Windows for clients that require more RAM.

Answer 3

You declare that you do not want to go to the database, but what about a database that uses AWE specifically ?

I have not tried this personally, but would consider using the products of this company for my own projects.

[Edit]: NexusDB is Delphi friendly: it came from the old Flash Flash Flash Turbopower development (but has since gone very far).

Answer 4

The problem with AWE is very similar to the old one based on DOS EMS and XMS - if you have ever used them. In principle, an address memory range is reserved, and then memory that is outside the address range is then mapped to the address range when necessary, and not displayed when it is no longer needed, allowing you to map another memory to the same addresses. Thus, most non-AWE structures or containers will not work in this scenario - perhaps a TMemoryStream descendant is easier to build. It should be easy enough to create a TList or the like that stores data in AWE-memory, it should keep track of where the data is actually stored, and if necessary, call it, and also adjust the addresses when the data is mapped to the address memory. I am not aware of any Delphi container library using AWE, and there is another problem: 32-bit desktop operating systems cannot use more than 4 GB of physical memory, a server version will be required, and the supported physical RAM is version dependent, see here for a complete list.

Answer 5

Assuming data is loaded once in bulk and is suitable for available memory, NexusDB AWE will be very fast. The database can be created as a database only in memory and then there will be no need for additional access to the hard disk during manipulation.

Answer 6

It sounds as if you guys could drop the current SQL database and switch to the 100% NexusDB + AWE solution.

(Or rather, discarding daily access to the SQL server and having an export / synchronization function that can write out any required NexusDB reporting data to the MSSQL report database.)

W

Answer 7

Your situation is similar to ours, our application uses a huge data file, which we store in a file with memory mapping. Files are about 750 MB, and we highlight data structures from them that use up to 1.5 GB of RAM.

We did not find a solution to limit 4 GB, except that some of them switched to FPC / Lazarus, until, unfortunately, Delphi is 64-bit. AWE does not work with Vista Home versions, nor could we get it to work with MMF.

You can try files with memory mapping using a sliding window, which means that you dynamically create representations of different fragments of the file depending on what part of it the application uses. It doesn't seem to work because you need the whole file in memory at once.