Attributes changed by on.the-fly lossy compression

[Markus Sabin]

[EDIT - moved this question from Installation to General]

Dear DCMTK-forum,

I have a question about changes which are applied to DICOM image attributes upon (lossy) on-the-fly compression. We can handle the DCMTK good enough to change the behaviour if needed, but we would like to understand the decisions made in development.

1. Image Type (0008,0008)
Originally the Image Type of our image (in this case: Computed Radiography) was set to "ORIGINAL\PRIMARY". After compression, it is set to "DERIVED". In our understanding, the Image Type shall have at least 2 values, so I would have expected that only the first component would be updated, resulting in an Image Type of DERIVED\PRIMARY.

(appears to be contradictory by itself, but as I understand PS3.3, C.7.6.1.1.2, this would be the correct value if the image is lossily compressed right after acquisition)

2. Derivation Code Sequence (0008,9215)
The sequence contains one item with the following code:

Code: Select all

    (0008,0100) SH [121327]                                 #   6, 1 CodeValue
    (0008,0102) SH [DCM]                                    #   4, 1 CodingSchemeDesignator
    (0008,0104) LO [Full fidelity image]                    #  20, 1 CodeMeaning

No Context ID is given, but I strongly suspect that this refers to PS3.16, CID 7205, in which an alternative code would be

Code: Select all

    (0008,0100) SH [121325]                                 #   6, 1 CodeValue
    (0008,0102) SH [DCM]                                    #   4, 1 CodingSchemeDesignator
    (0008,0104) LO [Lossy compressed image]                    #  20, 1 CodeMeaning

Since PS3.3, C.12.4.1.1 Derivation Description says that:

[...] Derivation Code Sequence (0008,9215) describe the way in which the image was derived [...]

I would think that code 121325 would describe "the way of derivation" more accurately. I suspect there are good reasons behind the behaviour that we are observing, but I would like to understand them.

Thoughts very welcome! Thank you!

[J. Riesmeier]

Dear Markus,

re. 1: Are you sure that the original Image Type was "ORIGINAL\PRIMARY" and not "ORIGINAL/PRIMARY" (or the like, i.e. without backslash)? I'm asking because DcmCodec::updateImageType() should make sure that the values after the first one are copied to the resulting dataset.

Re. 2: CID 7203 (Image Derivation​) would be the correct source to select a code from. E.g., (113040,DCM,"Lossy Compression​") would be appropriate in case of lossy image compression (e.g. JPEG baseline). This is also the way it is implemented.
In case of lossless image compression, I'm not sure whether the current implementation (using the code for "Full fidelity image" from CID 7205) is really appropriate. Maybe, Marco who implemented this part of the DCMTK can comment on this.

[Markus Sabin]

Hi Jörg,

thank you very much for your reply.

ad 1: Sorry for the confusion. I figured out that the Image Type was not included at all in this case. It was a CR image, for which Image Type is a Type 3 attribute, so the value changed from "<nothing>" to "DERIVED". It is not so nice that this renders the image invalid, but I think that this is the best choice the toolkit can take.

ad 2: Hmm...this does not seem to be the case here. After compression/transfer, we see the following header elements which we did not insert explicitly (BTW: The receiver is the DICOM Validation Tool, so it is more than unlikely that the receiver changes the attributes):

Code: Select all

# Dicom-Meta-Information-Header
# Used TransferSyntax: Little Endian Explicit
(0002,0010) UI =JPEGExtended:Process2+4                 #  22, 1 TransferSyntaxUID

# Dicom-Data-Set
# Used TransferSyntax: JPEG Extended, Process 2+4
(0008,0008) CS [DERIVED]                                #   8, 1 ImageType
(0008,2111) ST [Lossy compression with JPEG extended sequential 8 bit, IJG quality... # 102, 1 DerivationDescription
(0008,9215) SQ (Sequence with undefined length #=1)     # u/l, 1 DerivationCodeSequence
  (fffe,e000) na (Item with undefined length #=3)         # u/l, 1 Item
    (0008,0100) SH [121327]                                 #   6, 1 CodeValue
    (0008,0102) SH [DCM]                                    #   4, 1 CodingSchemeDesignator
    (0008,0104) LO [Full fidelity image]                    #  20, 1 CodeMeaning
  (fffe,e00d) na (ItemDelimitationItem)                   #   0, 0 ItemDelimitationItem
(fffe,e0dd) na (SequenceDelimitationItem)               #   0, 0 SequenceDelimitationItem

Anyway, we will look into this and see where this comes from. We will come back after we have more details. However, it may take some days until we can focus on that, thus this preliminary observation.

[Marco Eichelberg]

I have added this as issue #924 to the DCMTK issue tracker: https://support.dcmtk.org/redmine/issues/924.

I believe that this may be a left-over from the initial implementation of the DCMJPEG module, which always fed the image data through the DicomImage class prior to compression and could, thus, not guarantee that lossless compression was 100% lossless, because for example a color space conversion (with rounding error) might have taken place. This behaviour is now active only if you explicitly enable the "old pseudo-lossless codec", e.g. using dcmcjpeg --pseudo-lossless, which is mostly there for historical reasons. I guess that the decision to set the image type to derived and the DerivationCodeSequence to this value was a lame attempt to describe this behaviour: The image was not lossy compressed, but it was still somehow derived, and there was no better code in DICOM part 16. In any case, this should be removed.

What confuses me, though, is the DerivationDescription attribute that speaks of JPEG lossy extended sequential coding. Is it possible that your image has undergone multiple compression phases, or is there another bug that causes an incorrect derivation description to be generated?

[Markus Sabin]

Thank you very much for your assessment, Marco. Meanwhile we have investigated the issue in more detail.

We think that the problem is caused by DJCodecEncoder::updateDerivationDescription(), and we found discrepancies between the behavior of the command line tools (dcmcjpeg) and DCMTK integrated in an application which might be of interest for the DCMTK community.

The major difference between dcmcjpeg and a product using DCMTK is that dcmcjpeg uses exactly one set of compression parameters per execution and performs compression with exactly one output transfer syntax. An application may use different output transfer syntaxes in different situations. This is also why we were unable to reproduce the issue that the code "Full fidelity image" was also found for images compressed with JPEG baseline (lossy).

If the user sets the parameter --true-lossless for dcmcjpeg but chooses a lossy output transfer syntax, the parameter --true-lossless is forced to "false" by the parameter verification in dcmcjpeg. Our application which uses lossless and lossy JPEG only initializes the DcmCodecRegistry once setting the parameter to "true" - this is possible and makes sense to us. Obvioulsy, the parameter is then also "true" when the encoder performs lossy compression.

The method DJCodecEncoder::updateDerivationDescription() checks the parameter of the codec registry to determine whether it sets the derivation code sequence item to ("DCM", "121327", "Full fidelity image") or ("DCM", "113040", "Lossy Compression"). This decision is only based on the configuration of the codec registry. We think that it should also consider the selected output Transfer Syntax or the used coded respectively:

Code: Select all

    DJCodecParameter *djcp = OFconst_cast(DJCodecParameter*, cp);

    if (djcp->getTrueLosslessMode())
      result = DcmCodec::insertCodeSequence(dataset, DCM_DerivationCodeSequence, "DCM", "121327", "Full fidelity image");
    else // pseudo-lossless mode may also result in lossy compression
      result = DcmCodec::insertCodeSequence(dataset, DCM_DerivationCodeSequence, "DCM", "113040", "Lossy Compression");

We see two ways to solve this:
1. Change the implementation of DJCodecEncoder::updateDerivationDescription() in such a way that it considers the Transfer Syntax of the dataset (like dataset->getXFer().isLossy())
2. Add a polymorphic method to DJCodecEncoder which adds an appropriate item to the Derivation Code Sequence and override this method in derived encoder classes like it is implemented for createDerivationDescription().

We would be willing to fix this and provide a patch to OFFIS if you like and tell us which way to go.

Best Regards,

Markus

[Markus Sabin]

We have now changed the implementation of DJCodecEncoder::updateDerivationDescription() to:

Code: Select all

[...]
  OFCondition result = dataset->putAndInsertString(DCM_DerivationDescription, derivationDescription.c_str());
  if (result.good())
  {
	  // assume we can cast the codec parameter to what we need
	  DJCodecParameter *djcp = OFconst_cast(DJCodecParameter*, cp);

	  if(!isLosslessProcess() || !djcp->getTrueLosslessMode())
		  result = DcmCodec::insertCodeSequence(dataset, DCM_DerivationCodeSequence, "DCM", "113040", "Lossy Compression");
  }
[...]

It works fine for us. It would be nice to know if this change will be incorporated in the next DCMTK release, so that we can safely update to that version when it is released. As far as I can see, lossless compression does not count for "derivation", thus no appropriate code exists in CID 7203, and the Derivation Code Sequence shall not be included.

[Marco Eichelberg]

Thanks for the patch, which indeed fixes the problem. I have just committed the patch and closed DCMTK issue #924.