Memory leak with win32 storescp --fork

All other questions regarding DCMTK

Moderator: Moderator Team

Post Reply
Message
Author
rhinojunk
Posts: 10
Joined: Tue, 2006-03-14, 22:25

Memory leak with win32 storescp --fork

#1 Post by rhinojunk »

This is readily reproducible for me with the official 3.5.4 build on Windows Server 2003/XP, and on Windows 2000 Server using the build provided by Colby Dillion here: viewtopic.php?t=715

This seems to be related to the number of associations that get forked, and not the size of the pixel data. If left alone for 24 hours, storescp never releases the memory. Also, removing the "--fork" option results in zero growth.

The wierd part is that minimizing or unminimizing the storescp cmd window results in the memory being released.

If I can do any additional testing to help pin this down, let me know. Thanks!

1. Start storescp with forking support

Code: Select all

storescp.exe --fork -od Images --ignore 5432
2. Get storescp private byte usage.

Code: Select all

Name                Pid Pri Thd  Hnd   Priv        CPU Time    Elapsed Time 
storescp           6060   8   1   52   1624     0:00:00.015     0:00:03.187
3. Store a 128k MR 100 times inside the same association (1 fork).

Code: Select all

StoreSCU.exe --repeat 100 -aec STORESCP localhost 5432 Images-128KB\*
4. Get storescp private byte usage. (24 kilobytes growth)

Code: Select all

Name                Pid Pri Thd  Hnd   Priv        CPU Time    Elapsed Time 
storescp           6060   8   1   57   1648     0:00:00.015     0:00:04.140
5. Store a 128k MR 100 times inside seperate associations(100 forks). I used a batch file with a loop.

Code: Select all

StoreSCU.exe -aec STORESCP localhost 5432 Images-128KB\*
StoreSCU.exe -aec STORESCP localhost 5432 Images-128KB\*
StoreSCU.exe -aec STORESCP localhost 5432 Images-128KB\*
... 
6. Get storescp private byte usage. (1776 kilobytes growth)

Code: Select all

Name                Pid Pri Thd  Hnd   Priv        CPU Time    Elapsed Time 
storescp           6060   8   1  457   3424     0:00:00.062     0:00:09.265
7. Store a 10MB CR 100 times inside seperate associations (100 forks). I used a batch file with a loop.

Code: Select all

StoreSCU.exe -aec STORESCP localhost 5432 Images-10MB\*
StoreSCU.exe -aec STORESCP localhost 5432 Images-10MB\*
StoreSCU.exe -aec STORESCP localhost 5432 Images-10MB\*
... 
8. Get storescp private byte usage. (1764 kilobytes growth)

Code: Select all

Name                Pid Pri Thd  Hnd   Priv        CPU Time    Elapsed Time 
storescp           6060   8   1  857   5188     0:00:00.265     0:00:22.530
EDIT TO CHANGE: I first labeled the numbers above as bytes, but they are kilobytes. :oops:
Last edited by rhinojunk on Thu, 2006-06-08, 20:37, edited 1 time in total.

Marco Eichelberg
OFFIS DICOM Team
OFFIS DICOM Team
Posts: 1462
Joined: Tue, 2004-11-02, 17:22
Location: Oldenburg, Germany
Contact:

#2 Post by Marco Eichelberg »

This looks like a small resource leak indeed. My first guess is that this would happen in receiveTransportConnectionTCP() in dcmnet/libsrc/dul.cc, inside the code that creates the child process and passes the socket handle. It would be great if some of the windows API gurus in this forum could help looking for this. The only handle that does not seem to get closed in the case of a successful generation of a child process, is hChildStdInWriteDup. Can this be safely closed at a point in time when the child has possibly not yet read all data from the pipe?

Colby Dillion
Posts: 13
Joined: Tue, 2005-06-28, 16:48

#3 Post by Colby Dillion »

I am going to go ahead and post a reply before I lose my thoughts.

Once the child process has been created and you have passed all of the data that you need to pass you should be able to close hChildStdInWriteDup without any problems. This handle is never closed and that is an issue to look at. I am actually not sure why I never noticed this before.

If the resources get returned when the application is minimized or restored then it is possible that we are just dealing with a Windows peculiarity and not an actual bug (at least in dcmtk). I would be interested to know if the memory persists or if it disappears after 15 minutes or maybe even an hour. I have been wrong before so lets not count out an actual leak.

I will put together a patch for this so that we can have a fix before the next version of DCMTK is released. I need to remember to send Marco the half dozen other patches I have waiting also. We are actually in the process of converting from Perforce to Subversion for version control and when that happens I will open up our private DCMTK tree so it will be easy to track our patches. We have a few goodies hiding in there that people might be interested in including a (slightly buggy but fixable) JPEG 2000 codec (using OpenJPEG) and .NET SWIG interfaces for the dcmdata and dcmnet modules.

P.S. - What kind of memory constraints are you running on that you actually noticed 1.7kb of unreleased memory to begin with? ;)

rhinojunk
Posts: 10
Joined: Tue, 2006-03-14, 22:25

#4 Post by rhinojunk »

Colby Dillion wrote:I am going to go ahead and post a reply before I lose my thoughts.

Once the child process has been created and you have passed all of the data that you need to pass you should be able to close hChildStdInWriteDup without any problems. This handle is never closed and that is an issue to look at. I am actually not sure why I never noticed this before.

If the resources get returned when the application is minimized or restored then it is possible that we are just dealing with a Windows peculiarity and not an actual bug (at least in dcmtk). I would be interested to know if the memory persists or if it disappears after 15 minutes or maybe even an hour. I have been wrong before so lets not count out an actual leak.
I have let storescp sit idle for 24 hours after a leak, and the memory does not get released inside of that time frame.
Colby Dillion wrote:I will put together a patch for this so that we can have a fix before the next version of DCMTK is released. I need to remember to send Marco the half dozen other patches I have waiting also. We are actually in the process of converting from Perforce to Subversion for version control and when that happens I will open up our private DCMTK tree so it will be easy to track our patches. We have a few goodies hiding in there that people might be interested in including a (slightly buggy but fixable) JPEG 2000 codec (using OpenJPEG) and .NET SWIG interfaces for the dcmdata and dcmnet modules.

P.S. - What kind of memory constraints are you running on that you actually noticed 1.7kb of unreleased memory to begin with? ;)
Good point, I actually mis-reported the growth in the original post. Those numbers are kilobytes, not bytes. I'll correct that now.

rhinojunk
Posts: 10
Joined: Tue, 2006-03-14, 22:25

I think I found a big part of it...

#5 Post by rhinojunk »

I decided it was time to get my hands dirty, so I installed Visual Studio and started digging. I'm not a coder by trade, so there may be better ways to address these leaks.

1. Mr. Colby Dillion was correct, in that there are a number of handles which need to be closed in the area he describes. I was seeing a leak of 4handles/association, and here's what I changed to fix that:

PS - From what I gather using google, the closesocket() calls can cause trouble with Windows 95 if the child isn't done using the socket yet. Unfortunatly using CloseHandle() on the accept socket handles resulted in odd behavior and continued handle leaking.

dul.cxx ~line 1700 before:

Code: Select all

            // PROCESS_INFORMATION pi now contains various handles for the new process.
			// Now that we have a handle to the new process, we can duplicate the
			// socket handle into the new child process.
			if (DuplicateHandle(GetCurrentProcess(), (HANDLE)sock, pi.hProcess, 
                &childSocketHandle, 0, TRUE, DUPLICATE_CLOSE_SOURCE)) 
            {
                // close handles in PROCESS_INFORMATION structure
				// and our local copy of the socket handle.
                CloseHandle(pi.hProcess);
                CloseHandle(pi.hThread);
                CloseHandle((HANDLE)sock);

				// send number of socket handle in child process over anonymous pipe
                DWORD bytesWritten;
                char buf5[20];
                sprintf(buf5,"%i",(int)childSocketHandle);
                if (!WriteFile(hChildStdInWriteDup, buf5, strlen(buf5)+1, &bytesWritten, NULL)) 
				{
                    CloseHandle(hChildStdInWriteDup);
                    return makeDcmnetCondition (DULC_CANNOTFORK, OF_error, "error while writing to anonymous pipe");
                }

				// return OF_ok status code DULC_FORKEDCHILD with descriptive text
                char buf4[256];
                sprintf(buf4, "new child process started with pid %i, socketHandle %i", 
                  OFstatic_cast(int, pi.dwProcessId), 
                  (int)childSocketHandle);
                return makeDcmnetCondition (DULC_FORKEDCHILD, OF_ok, buf4);
            }
            else 
            {
                // unable to duplicate handle. Close handles nevertheless
				// to avoid resource leak.
                CloseHandle(pi.hProcess);
                CloseHandle(pi.hThread);
                CloseHandle((HANDLE)sock);
                return makeDcmnetCondition (DULC_CANNOTFORK, OF_error, "error while duplicating socket handle");
            }
dul.cxx ~line 1700 after:

Code: Select all

            // PROCESS_INFORMATION pi now contains various handles for the new process.
			// Now that we have a handle to the new process, we can duplicate the
			// socket handle into the new child process.
			HANDLE ParentProcessHandle = OpenProcess(PROCESS_ALL_ACCESS, FALSE, GetCurrentProcessId()); // (Colby Dillon) fix for Win2k support
			if (DuplicateHandle(ParentProcessHandle, (HANDLE)sock, pi.hProcess,
                &childSocketHandle, 0, TRUE, DUPLICATE_SAME_ACCESS)) // (Colby Dillon) fix for Win2k support
            {
                // close handles in PROCESS_INFORMATION structure
				// and our local copy of the socket handle.
                CloseHandle(pi.hProcess);
                CloseHandle(pi.hThread);
                CloseHandle(ParentProcessHandle); // (Colby Dillon) fix for Win2k support
				closesocket((SOCKET)sock); // (rhinojunk) fixes handle leak

				// send number of socket handle in child process over anonymous pipe
                DWORD bytesWritten;
                char buf5[20];
                sprintf(buf5,"%i",(int)childSocketHandle);
                if (!WriteFile(hChildStdInWriteDup, buf5, strlen(buf5)+1, &bytesWritten, NULL)) 
				{
                    CloseHandle(hChildStdInWriteDup);
					CloseHandle(hChildStdInRead); // (rhinojunk) fixes handle leak
					closesocket((SOCKET)childSocketHandle); // (rhinojunk) fixes handle leak
                    return makeDcmnetCondition (DULC_CANNOTFORK, OF_error, "error while writing to anonymous pipe");
                }
                
				CloseHandle(hChildStdInWriteDup); // (rhinojunk) fixes handle leak
				CloseHandle(hChildStdInRead); // (rhinojunk) fixes handle leak

				// return OF_ok status code DULC_FORKEDCHILD with descriptive text
                char buf4[256];
                sprintf(buf4, "new child process started with pid %i, socketHandle %i", 
                  OFstatic_cast(int, pi.dwProcessId), 
                  (int)childSocketHandle);
				closesocket((SOCKET)childSocketHandle); // (rhinojunk) fixes handle leak
                return makeDcmnetCondition (DULC_FORKEDCHILD, OF_ok, buf4);
            }
            else 
            {
                // unable to duplicate handle. Close handles nevertheless
				// to avoid resource leak.
                CloseHandle(pi.hProcess);
                CloseHandle(pi.hThread);
				CloseHandle(ParentProcessHandle); // (Colby Dillon) fix for Win2k support
                closesocket((SOCKET)sock); // (rhinojunk) fixes handle leak
				CloseHandle(hChildStdInWriteDup); // (rhinojunk) fixes handle leak
				CloseHandle(hChildStdInRead); // (rhinojunk) fixes handle leak
                return makeDcmnetCondition (DULC_CANNOTFORK, OF_error, "error while duplicating socket handle");
            }

2. Most of the memory leak appears to be caused when the DUL_ReceiveAssociationRQ function calls createAssociationKey creating a PRIVATE_ASSOCIATIONKEY structure which never gets released. My fix is as follows:

dul.cxx ~line 597 before:

Code: Select all

cond = receiveTransportConnection(network, block, timeout, params, association);

if (cond.bad() || (cond.code() == DULC_FORKEDCHILD))
    return cond;
dul.cxx ~line 597 after:

Code: Select all

cond = receiveTransportConnection(network, block, timeout, params, association);

if (cond.bad() || (cond.code() == DULC_FORKEDCHILD))
{
    destroyAssociationKey(association); // (rhinojunk) otherwise association leaks ~17KB/association
    return cond;
}
The result is that I'm seeing a leak of only about 0.6KB/association instead of 18KB/association.

rhinojunk
Posts: 10
Joined: Tue, 2006-03-14, 22:25

Found 1 more

#6 Post by rhinojunk »

I found another ~660 bytes/association leaking in ASC_receiveAssociation() very similar to the leak in DUL_ReceiveAssociationRQ(). This time it's the T_ASC_Parameters params struct getting malloc'd but never released. The fix is very similar as well. With this change I'm no longer seeing significant growth.

assoc.cxx ~line 1554 before:

Code: Select all

    if (cond.code() == DULC_FORKEDCHILD) return cond;
assoc.cxx ~line 1554 after:

Code: Select all

    if (cond.code() == DULC_FORKEDCHILD)
	{
		ASC_destroyAssociationParameters(&params);
		return cond;
	}

Marco Eichelberg
OFFIS DICOM Team
OFFIS DICOM Team
Posts: 1462
Joined: Tue, 2004-11-02, 17:22
Location: Oldenburg, Germany
Contact:

#7 Post by Marco Eichelberg »

Thanks a lot for this. I'll make sure that your fixes get merged into our main CVS and are included in the next release.

Marco Eichelberg
OFFIS DICOM Team
OFFIS DICOM Team
Posts: 1462
Joined: Tue, 2004-11-02, 17:22
Location: Oldenburg, Germany
Contact:

#8 Post by Marco Eichelberg »

The fix has been committed to CVS today and will be part of the next DCMTK snapshot.

Post Reply

Who is online

Users browsing this forum: Bing [Bot] and 0 guests