Memory Mapped file support

All other questions regarding DCMTK

Moderator: Moderator Team

Post Reply
Message
Author
anilta
Posts: 60
Joined: Thu, 2014-04-10, 08:50

Memory Mapped file support

#1 Post by anilta »

Hi, we are creating a huge DICOM image (Whole Slide DICOM image IOD), which would be bigger than the virtual memory available. Does DCMTK support or have classes which can help (similar to Memory Mapped Files of Windows) ?

Michael Onken
DCMTK Developer
Posts: 2048
Joined: Fri, 2004-11-05, 13:47
Location: Oldenburg, Germany
Contact:

Re: Memory Mapped file support

#2 Post by Michael Onken »

Hi,

unfortunately there is no such class in DCMTK.

However, I think you could use DcmElement::createValueFromTempFile() in order to insert pixel data from a pixel data file stored on disk. This means you would write all your pixel data to disk in the format that it will later have inside the Pixel Data element and then use this method to insert it into the current file in memory, and then write that file normally using saveFile() to disk. As far as I remember, when writing, the "pixel data file" is not actually fully loaded into memory, but instead, is copied buffer-wise from the source (pixel data file) to the destination file (written with saveFile()).

I never used it myself, so maybe one of my colleagues can correct me if necessary.

You can find an example of createValueFromTempFile() in the dcmodify tool (options -if and -mf) which is then actually handled in dcmdata/libsrc/mdfdsman.cc. Maybe you can just try to insert some GB of pixel data using dcmodify -if ... and have an eye on memory usage to see whether it works in general.

BR, Michael

Michael Onken
DCMTK Developer
Posts: 2048
Joined: Fri, 2004-11-05, 13:47
Location: Oldenburg, Germany
Contact:

Re: Memory Mapped file support

#3 Post by Michael Onken »

P.S: Well, I tried this and actually in dcmodify this does not work (loads fully into memory instead). Anyway, maybe Jörg or Marco know the best way to handle your problem.

J. Riesmeier
DCMTK Developer
Posts: 2501
Joined: Tue, 2011-05-03, 14:38
Location: Oldenburg, Germany
Contact:

Re: Memory Mapped file support

#4 Post by J. Riesmeier »

To the OP: Since WSI objects are typically stored as compressed multi-frame images where each frame represents a tile of the entire images, you could use the createValueFromTempFile() method to specify the pixel data for each tile individually. Alternatively, you could create the pixel data of the tiles on demand, which requires to derive a subclass from the DcmPixeltem class.

anilta
Posts: 60
Joined: Thu, 2014-04-10, 08:50

Re: Memory Mapped file support

#5 Post by anilta »

Thanks Jörg & Michael. This gives some idea on how to proceed.

Our workflow is, the client requests to transfer a huge pathology image to a DICOM server in Whole Slide Image IOD.
Client gives all the tiles data to DCMTK DLL, then DCMTK DLL collects all the tiles data, add them to the target DICOM image.
This DICOM image (DcmDataset) is created in memory, and once the creation is done, the memory copy is transferred to DICOM server.
So both the tile data are in memory and also the target DICOM image (DcmDataset) is also in memory.
Hence we are trying to achieve the memory requirement for this using memory mapped files.

J. Riesmeier
DCMTK Developer
Posts: 2501
Joined: Tue, 2011-05-03, 14:38
Location: Oldenburg, Germany
Contact:

Re: Memory Mapped file support

#6 Post by J. Riesmeier »

If all data should reside in main memory, using the "create tile pixel data on demand" would be my first choice.

anilta
Posts: 60
Joined: Thu, 2014-04-10, 08:50

Re: Memory Mapped file support

#7 Post by anilta »

Hi Jörg,

I am not able to get an complete picture on my requirement 'create the whole image in memory and send this to DICOM server'. My apologies.

You mean, after the store is initiated, create the image, add tile by tile to the DICOM image and and send these chunks of image data to DICOM server ?

J. Riesmeier
DCMTK Developer
Posts: 2501
Joined: Tue, 2011-05-03, 14:38
Location: Oldenburg, Germany
Contact:

Re: Memory Mapped file support

#8 Post by J. Riesmeier »

I mean you could derive a subclass from DcmPixelItem which stores information on how to retrieve the pixel data for the particular tile from the source, but only creates the byte stream when needed (i.e. on demand), which is when it is transferred over the network (or written to file).

This is the short story of doing it. I think the long story is too complex to explain in a public discussion forum like this. Bottom line: it is possibly with DCMTK to reduce the memory footprint, but it does not work "out of the box".

PS: I already did this for one of my customers, so I know that it works.

Post Reply

Who is online

Users browsing this forum: Ahrefs [Bot], Bing [Bot], Google [Bot] and 1 guest