memory leak while encoding

[gayathri.emd]

Hi,
In my windows application i am processing(loading ->compress->save) US Multiframe file(1021 frames, size-1.5gb, decompressed file) 100 times.
below code is executed.



Code Snippet:

if (filePtr.loadFile(dicom_file_path).good())
{
DcmDataset *dataset = filePtr.getDataset();
E_TransferSyntax t_xfer = EXS_JPEGProcess14SV1;
E_EncodingType encoding_type = EET_ExplicitLength;
std::string fullpath = "D:\\test\\output\\" + count;
OFCOndition status = dataset->chooseRepresentation(t_xfer, NULL);
status = filePtr.saveFile(fullpath.c_str(), t_xfer, encoding_type, EGL_recalcGL, EPD_withoutPadding, 0, 0, EWM_fileformat);
}

experiencing 450mb memory leak after execution completion(private byte not decreasing).


Note dcmtk version: 3.6.6

[Michael Onken]

Hi,

can you use a newer version of DCMTK (3.6.7, at least, of preferably current source code from https://github.com/DCMTK/dcmtk/)?

I used a leak checker (on Linux, though) on your slightly modified source code (using current source of DCMTK). It performs the compression 10 times on a Enhanced XA with 140 MB uncompressed (don't have Enhanced US here but it does not make a difference):

Code: Select all

#include "dcmtk/config/osconfig.h"   // make sure OS specific configuration is included first
#include "dcmtk/dcmdata/dcfilefo.h"

int main(int argc, char *argv[])
{
  DcmFileFormat ff;

  static size_t count = 1;
  while (ff.loadFile("/tmp/1.dcm").good() && (count <= 10))
  {
    std::cout << "Processing file " << count << std::endl;
    DcmDataset *dataset = ff.getDataset();
    E_TransferSyntax t_xfer = EXS_JPEGProcess14SV1;
    E_EncodingType encoding_type = EET_ExplicitLength;
    std::string fullpath = "/tmp/output_" + std::to_string(count);
    OFCondition status = dataset->chooseRepresentation(t_xfer, NULL);
    status = ff.saveFile(fullpath.c_str(), t_xfer, encoding_type, EGL_recalcGL, EPD_withoutPadding, 0, 0, EWM_fileformat);
    count++;
  }
}

The checker reports no possible leaks (i.e. gives guarantee):

Code: Select all

❯ valgrind --leak-check=full bin/checkleak 
==228929== Memcheck, a memory error detector
==228929== Copyright (C) 2002-2022, and GNU GPL'd, by Julian Seward et al.
==228929== Using Valgrind-3.22.0 and LibVEX; rerun with -h for copyright info
==228929== Command: bin/dcmodify
==228929== 
Processing file 1
Processing file 2
Processing file 3
Processing file 4
Processing file 5
Processing file 6
Processing file 7
Processing file 8
Processing file 9
Processing file 10
==228929== 
==228929== HEAP SUMMARY:
==228929==     in use at exit: 0 bytes in 0 blocks
==228929==   total heap usage: 370,620 allocs, 370,620 frees, 13,902,619 bytes allocated
==228929== 
==228929== All heap blocks were freed -- no leaks are possible
==228929== 
==228929== For lists of detected and suppressed errors, rerun with: -s
==228929== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

Maybe you could try with the above code and see whether the problem is maybe in part of your code that you did not paste. I don't think there is Windows-specific code involved the code snippet you presented. I will also try with Windows (and not having a good leak checker there watch memory in task manager) but DCMTK build takes (much) longer there...

Overall I cannot see where there is a leak right now in DCMTK.

BR Michael

[Michael Onken]

P.S: I checked now on Windows and at least in Task Manager i could not see rising memory consumption (converted 100 times on Windows).

[gayathri.emd]

Hi,
can you use a newer version of DCMTK (3.6.7, at least, of preferably current source code from https://github.com/DCMTK/dcmtk/)?


i will check with latest dcmtk and update you.

[Michael Onken]

Please also post a full compiling code example that demonstrates the issue.

Thanks!

[gayathri.emd]

Hi,
i have tested below code with latest dcmtk(master branch).Still, i am facing memory leak issue.

Code: Select all

#include "dcmtk/config/osconfig.h"   
#include "dcmtk/dcmdata/dcfilefo.h"
#include "dcmtk\dcmjpeg\djdecode.h"
#include "dcmtk\dcmjpeg\djencode.h"
int main(int argc, char *argv[])
{
	DJDecoderRegistration::registerCodecs();
	DJEncoderRegistration::registerCodecs();
	
	

	static size_t count = 1;
	while ( count <= 200)
	{
		DcmFileFormat ff;
		if (ff.loadFile("F:\\test.dcm").good())
		{
			std::cout << "Processing file " << count << std::endl;
			DcmDataset *dataset = ff.getDataset();
			E_TransferSyntax t_xfer = EXS_JPEGProcess14SV1;
			E_EncodingType encoding_type = EET_ExplicitLength;
			std::string fullpath = "D:\\test\\output_" + std::to_string(count);
			OFCondition status = dataset->chooseRepresentation(t_xfer, NULL);
			status = ff.saveFile(fullpath.c_str(), t_xfer, encoding_type, EGL_recalcGL, EPD_withoutPadding, 0, 0, EWM_fileformat);
			count++;
		}
		
	}
	system("pause");
	DJDecoderRegistration::cleanup();
	DJEncoderRegistration::cleanup();

}

[Michael Onken]

I'll check and let you know (sorry, I realized my last code snippet did not compress at all and I know make sure error status is checked).

[Michael Onken]

Ok, here is my result of further investigation:

I tried on Windows (Win 10, VS 2022, 64 bit build) and Linux (Ubuntu 23, gcc 13.2.0, 64 bit build ). On Linux the leak checker still reports, that no memory leaks are possible. I value this very high since valgrind is 100% reliable in my experience and knowledge.

However: During execution of the program, I see that on both OS memory grows from initially 650 KB (before loading any file) to about 300 MB max. This maximum reached quickly and then not growing with further files. My input file is ~190 MB so 300 MB looks quite reasonable to me.

On exit, after cleanup of codecs, the process still has around 3 MB allocated (Windows), and even 60 MB (Linux). This is substantially more than the initial 650 KB. However, I think this is not due to leaks in DCMTK but instead the OS memory management that does return memory to the OS automatically (to make it available to other process), unless there is a strong need (i.e. running out of memory) for it. This seems to be true in particular where small amounts of memory are allocated. I guess that is what happens in the (C) libjpeg library.

How much leaking memory do you experience on your Windows system, btw?

Here are some hints about that "lazy" memory management of the OS on Stackoverflow (1,2,3).

So unless you find other evidence that there is memory leaking (maybe you have a leak check tool for Windows?), then I would not see a reason or way to investigate this further.

BR Michael

[gayathri.emd]

Hi,
I am getting memory leak around 320mb while using US(1.5 GB -attached )file in paused state.

Sample US File:
https://drive.google.com/drive/folders/ ... ctUycG1Wkj

1.Please run utility 2-3 times. Sometime i am not able to reproduce memory leak . Sometimes memory leak occured.

[J. Riesmeier]

Calling "delete filePtr.getDataset();" is certainly not a good idea, as the ownership of the internally managed dataset pointer is still with the DcmFileFormat instance. Instead, you should either call "delete filePtr.getAndRemoveDataset();" or "filePtr.clear();".

By the way, "filePtr" does not refer to a pointer, so the name is misleading.

[gayathri.emd]

Hi,

Tried "delete filePtr.getAndRemoveDataset();" / "filePtr.clear(). But still facing memory leak around 460MB in windows

Code: Select all

include "dcmtk/config/osconfig.h"   
#include "dcmtk/dcmdata/dcfilefo.h"
#include "dcmtk\dcmjpeg\djdecode.h"
#include "dcmtk\dcmjpeg\djencode.h"
int main(int argc, char *argv[])
{
	DJDecoderRegistration::registerCodecs();
	DJEncoderRegistration::registerCodecs();
	
	

	static size_t count = 1;
	while ( count <= 200)
	{
		DcmFileFormat ff;
		if (ff.loadFile("F:\\shared_temp\\DICOM_RECEIVE\\INPUT\\2023\\11\\30\\17\\16\\1.2.276.0.7230010.3.1.4.0.2751653766.33684.1701344870.237").good())
		{
			std::cout << "Processing file " << count << std::endl;
			DcmDataset *dataset = ff.getDataset();
			E_TransferSyntax t_xfer = EXS_JPEGProcess14SV1;
			E_EncodingType encoding_type = EET_ExplicitLength;
			std::string fullpath = "D:\\shared_temp\\shared\\output_" + std::to_string(count);
			OFCondition status = dataset->chooseRepresentation(t_xfer, NULL);
			status = ff.saveFile(fullpath.c_str(), t_xfer, encoding_type, EGL_recalcGL, EPD_withoutPadding, 0, 0, EWM_fileformat);
			count++;
		}
		delete ff.getAndRemoveDataset();

	}
	system("pause");
	DJDecoderRegistration::cleanup();
	DJEncoderRegistration::cleanup();

}

.

[Michael Onken]

Hi,

give me some time to check this again using the file you provided. Note that the posting of user "hana" was probably automatically generated (as well as other responses by the user) and thus we removed it. The answer did not making sense at all.

BR Michael

[gayathri.emd]

Hi,
Okay Michael .Thanks for your Response.