dcmj2pnm segfault with large multiframe US

All other questions regarding DCMTK

Moderator: Moderator Team

Message
Author
jberger
Posts: 30
Joined: Thu, 2015-09-03, 09:36

dcmj2pnm segfault with large multiframe US

#1 Post by jberger »

Hello,

a while ago I had a similar problem on Windows - using 64bit binaries solved the problem. Now it appears on Mac OS. The DICOM source file contains >1800 frames at 30fps and 1280x960. This results in a segmentation fault both on v3.6.0 and 3.6.2. Splitting the frames would be possible, but is it needed.

Hardware: Macbook Pro, 16GB RAM, 2,8 GHz Intel Core i7. Could this be a memory exhausting problem?

Code: Select all

macbook:~ jb$ ls -lh ./A0000
-rwxr-xr-x@ 1 jb  staff   222M 25 Sep 23:07 ./A0000
macbook:~ jb$ file ./A0000
./A0000: DICOM medical imaging data
macbook:~ jb$./dcmj2pnm +Fa +op -O --log-level trace A0000 ../mf
D: Start Of Frame 0xc0: width=1280, height=960, components=3
D:     Component 0: 2hx1v q=0
D:     Component 1: 1hx1v q=1
D:     Component 2: 1hx1v q=1
D: Define Huffman Table 0x00
T:           0   1   5   1   1   1   1   1
T:           1   0   0   0   0   0   0   0
D: Define Huffman Table 0x10
T:           0   2   1   3   3   2   4   3
T:           5   5   4   4   0   0   1 125
D: Define Huffman Table 0x01
T:           0   3   1   1   1   1   1   1
T:           1   1   1   0   0   0   0   0
D: Define Huffman Table 0x11
T:           0   2   1   2   4   4   3   4
T:           7   5   4   4   0   1   2 119
D: Start Of Scan: 3 components
D:     Component 0: dc=0 ac=0
D:     Component 1: dc=1 ac=1
D:     Component 2: dc=1 ac=1
D:   Ss=0, Se=63, Ah=0, Al=0
Segmentation fault: 11
Many thanks for any hints.


Jens

Michael Onken
DCMTK Developer
Posts: 2048
Joined: Fri, 2004-11-05, 13:47
Location: Oldenburg, Germany
Contact:

Re: dcmj2pnm segfault with large multiframe US

#2 Post by Michael Onken »

Hi Jens,

can you provide the image, maybe anonymized? If possible, you can send it (via download link) to onken ät open-connections dot de, I'll share it with the rest of the (small) DCMTK team. It seems the segfault happens in the 3rd-party JPEG decoder we integrated into DCMTK :/

Best,
Michael

jberger
Posts: 30
Joined: Thu, 2015-09-03, 09:36

Re: dcmj2pnm segfault with large multiframe US

#3 Post by jberger »

Hi Michael,

the download link has been sent.

Many thanks

Jens

Marco Eichelberg
OFFIS DICOM Team
OFFIS DICOM Team
Posts: 1437
Joined: Tue, 2004-11-02, 17:22
Location: Oldenburg, Germany
Contact:

Re: dcmj2pnm segfault with large multiframe US

#4 Post by Marco Eichelberg »

It seems that you have found a bug in DCMTK here. The sample image you sent cannot be converted to an uncompressed DICOM format because the pixel data would be larger than 4 GBytes (1802 frames, 1280x960, RGB results in ca. 6.3 GBytes uncompressed). The dcmj2pnm tool is not too clever when it comes to handling multi-frame images, it will attempt to decompress the complete multi-frame image, in memory. This process should actually fail with an error message that the image is too large for that, but it seems that some calculation overflows or gets cut off through a typecast. So DCMTK allocates unsufficient memory and then happily starts filling that buffer with image information until the point (around 2 GBytes) where it starts writing beyond the buffer size and causes a segfault. We will fix that, but that will not help you much in this specific case, because instead of crashing dcmj2pnm will just refuse to process the image.

Marco Eichelberg
OFFIS DICOM Team
OFFIS DICOM Team
Posts: 1437
Joined: Tue, 2004-11-02, 17:22
Location: Oldenburg, Germany
Contact:

Re: dcmj2pnm segfault with large multiframe US

#5 Post by Marco Eichelberg »

The bug has been registered in our bug tracker as issue #793: http://support.dcmtk.org/redmine/issues/793

jberger
Posts: 30
Joined: Thu, 2015-09-03, 09:36

Re: dcmj2pnm segfault with large multiframe US

#6 Post by jberger »

Hello Marco,
Marco Eichelberg wrote:It seems that you have found a bug in DCMTK here. The sample image you sent cannot be converted to an uncompressed DICOM format because the pixel data would be larger than 4 GBytes (1802 frames, 1280x960, RGB results in ca. 6.3 GBytes uncompressed). The dcmj2pnm tool is not too clever when it comes to handling multi-frame images, it will attempt to decompress the complete multi-frame image, in memory. This process should actually fail with an error message that the image is too large for that, but it seems that some calculation overflows or gets cut off through a typecast. So DCMTK allocates unsufficient memory and then happily starts filling that buffer with image information until the point (around 2 GBytes) where it starts writing beyond the buffer size and causes a segfault. We will fix that, but that will not help you much in this specific case, because instead of crashing dcmj2pnm will just refuse to process the image.
Is there another or even better tool I should use for this purpose? Is there any workaround you know of?

I need access to each frame of these multiframe images in order to anonymize them. In the meanwhile I would try to extract the frames in parts of ca. 1000 Frames, which seems to work, but I have to re-enumerate the .ppm files to re-concatenate them ...

Do you have a better idea to anonymize multiframe US images?

Thanks for your support anyway,

Jens

J. Riesmeier
DCMTK Developer
Posts: 2501
Joined: Tue, 2011-05-03, 14:38
Location: Oldenburg, Germany
Contact:

Re: dcmj2pnm segfault with large multiframe US

#7 Post by J. Riesmeier »

I would write a little program for this purpose. The DicomImage class, which is used by dcmj2pnm, is able to access only parts of the pixel data element if initialized appropriately (see this Howto).

However, are you sure that using rendered pixel data (as created by dcmj2pnm and the DicomImage class) is a good idea when creating a DICOM image as output? How do you ensure that the describing header information still fits the value of the Pixel Data element?

Have you already checked David Clunie's "DicomCleaner"?

jberger
Posts: 30
Joined: Thu, 2015-09-03, 09:36

Re: dcmj2pnm segfault with large multiframe US

#8 Post by jberger »

My little 'software' consists of some shell scripts. I'm radiologist with a little bit of experience in shell programming.

Unfortunately, I have no experience with programming C.

The created output using ffmpeg reading ppm output of dcmj2pnm is not DICOM!
Target format is quicktime for Mac OS, or avi with wmv2 codec for Windows. The goal is to have ANONYMIZED, most compatible movie files running smoothly in Apple Keynote or MS Powerpoint.

The main script (dcm2mov.sh) extracts the multiframe pixel data, generates one or multiple overlay images (depending on ultrasound machine) to
remove the name of the patient from the US image file and concatenates the ppm files with ffmpeg to quicktime ...

David Clunies DICOMCleaner does not help in this case, because the patient name is coded in the pixel data.

Another (simpler) approach is highly appreciated!

Jens

J. Riesmeier
DCMTK Developer
Posts: 2501
Joined: Tue, 2011-05-03, 14:38
Location: Oldenburg, Germany
Contact:

Re: dcmj2pnm segfault with large multiframe US

#9 Post by J. Riesmeier »

Did you miss this section? http://www.dclunie.com/pixelmed/softwar ... l#Blackout

I have no Mac but OsiriX might also provide a solution (including export to MPEG4 and Quicktime).

jberger
Posts: 30
Joined: Thu, 2015-09-03, 09:36

Re: dcmj2pnm segfault with large multiframe US

#10 Post by jberger »

Yes, I missed this section. Sorry.

My 'software' is working since five years for a bunch of people very well, for sone users with storescp support and automagic conversion after receiving the data from the ultrasound machine via ethernet. There is no need to install Java. It is absolutely no user interaction required for anonymization and conversion, the user will find the movie files on their harddisk ready for insert into Keynote. The generated overlays are not recognizable by the naked eye. Most users of my software are running Mac OS X. They are physicians with lots of exams per day needed to be converted without any interaction.

OSiriX is a nice software, but here is also too much user interaction needed.

My approach seems to be not the most elegant, but I found no better yet for this purpose. Still looking for a more efficient way ...

Many thanks,

Jens

J. Riesmeier
DCMTK Developer
Posts: 2501
Joined: Tue, 2011-05-03, 14:38
Location: Oldenburg, Germany
Contact:

Re: dcmj2pnm segfault with large multiframe US

#11 Post by J. Riesmeier »

A more efficient way would probably require programming (see above). For example, I did this many years ago for the dcm2avi tool and underlying library (Windows only, though).

jberger
Posts: 30
Joined: Thu, 2015-09-03, 09:36

Re: dcmj2pnm segfault with large multiframe US

#12 Post by jberger »

I really would like to transform my shell scripts into a much more efficient program using a mixture of storescp, dcmdump, dcm2jpnm, convert, identify, ffprobe and ffmpeg, but this would require learning C, which would cost me years. Unfortunately I'm fully absorbed with my daily work as a radiologist. Shell programming is much more simple for me and does the job.

Until now.

Wouldn't you like to write such a program? ;-)

Best,

Jens

J. Riesmeier
DCMTK Developer
Posts: 2501
Joined: Tue, 2011-05-03, 14:38
Location: Oldenburg, Germany
Contact:

Re: dcmj2pnm segfault with large multiframe US

#13 Post by J. Riesmeier »

Wouldn't you like to write such a program? ;-)
Sure, but not for free ;-) See: https://www.jriesmeier.com/development/

jberger
Posts: 30
Joined: Thu, 2015-09-03, 09:36

Re: dcmj2pnm segfault with large multiframe US

#14 Post by jberger »

If I understand the bug tracker ticket right, buffer size > 4 GByte is not permitted in DICOM? Does this mean, the problem with such a large DICOM multiframe file is not to solve?

Why is it needed to decompress all images into the RAM?

Wouldn't it be possible to write the multiframe ppm file directly to the harddisk?

Jens

J. Riesmeier
DCMTK Developer
Posts: 2501
Joined: Tue, 2011-05-03, 14:38
Location: Oldenburg, Germany
Contact:

Re: dcmj2pnm segfault with large multiframe US

#15 Post by J. Riesmeier »

If I understand the bug tracker ticket right, buffer size > 4 GByte is not permitted in DICOM? Does this mean, the problem with such a large DICOM multiframe file is not to solve?
Yes, DICOM does not allow element values to be larger than 4 GB - 1, but no, this does not mean that the problem cannot be solved. It's just the reason for the internal limitation of the dcmj2pnm tool (and its underlying library), i.e. the segfault is also caused by the current implementation (which stores the decompressed pixel data in a DICOM element value as described above).
Why is it needed to decompress all images into the RAM?
It is not needed. This is just the way it was implemented (15 or 20 years ago, when the DICOM multi-frame images were not that large).
Wouldn't it be possible to write the multiframe ppm file directly to the harddisk?
There is no multi-frame PPM file (as far as I know) and it's not the PPM file that causes the segfault (see above).

I would rather recommend to implement the following command line option from dcm2avi also for dcm[j]2pnm:

Code: Select all

memory handling:

  -fm   --frames-in-memory  [f]rame count: integer
          number of frames in memory (default: all)

Post Reply

Who is online

Users browsing this forum: Ahrefs [Bot] and 1 guest