How to improve movescu performance?

[tchiang]

We use movescu inside a while loop in a perl script to pre-fetch priors. In one of our tests, it took more than 20 hours to run. Among the processed studies, there are 21 studies with Status = Success studies, 16 with Status = Warning (SubOperationsCompleteOneOrMoreFailures) and 25 with Status = Refused (OutOfResourcesSubOperations). I found that it spent a lot of time in Status = Pending.
We have to improve the speed because the time it took is unacceptable. There are several approaches I have in my mind:

• 1. What is OutOfResourcesSubOperations error? Is it because the images are off-line? Can we detect it (for example using C-FIND) and skip the study or abandon the processing at early stage (before we spend a lot of time waiting for every image to be examined)?
  2. Can we configure movescu to abandon processing if it has been pending for a specific amount of time (say 5 minutes)? Is –to option for this purpose? (In one of my testing case having only 4 images, it spent almost one hour in Status = Pending.)
  3. Can we configure movescu to abandon processing if more than a specific percentage of the images (say 50%) fail? Can we use –pi or –pr option for this purpose? (Currently, it spends a long time to examine till the last images before it sends Status = Refused signal. It is extremely time wasting for some studies with hundreds of images.)
  4. Execute movescu calls in the background (meaning add & at the end of the command line.) We don’t have to wait for the previous movescu to finish before we start another movescu. Unfortunately, this approach may reach beyond the maximum number of the association allowed on some of our DICOM devices.

Any other idea or thoughts are welcome too.
Thanks a lot.

[Michael Onken]

Hi,

that huge time amount seems to be very _very_ suspect. I guess there are other problems than just perfomance issues of the movescu code itself (if you do not move huge amounts of Gigabytes)

1. What is OutOfResourcesSubOperations error?

That usually means that the PACS is not able to send the image (sub operation) to the receiver because of being "out of resources". You have to check the conformance statement of the PACS to find out what that means for your specific PACS implementation.

2. Can we configure movescu to abandon processing if it has been pending for a specific amount of time (say 5 minutes)?

I'm not aware of any that fit your purpose - you could try aroudn with the options

Code: Select all

 
 -to   --timeout  [s]econds: integer (default: unlimited)
          timeout for connection requests

  -ta   --acse-timeout  [s]econds: integer (default: 30)
          timeout for ACSE messages

  -td   --dimse-timeout  [s]econds: integer (default: unlimited)
          timeout for DIMSE messages

3. Can we configure movescu to abandon processing if more than a specific percentage of the images (say 50%) fail?

No.

4. Execute movescu calls in the background (meaning add & at the end of the command line.) We don’t have to wait for the previous movescu to finish before we start another movescu.

You are free starting as many movescu calls as you want to (be sure that if you use movescu also as receiver to choose different receiver ports) - but you cannot tell movescu directly to start more than one process or thread, so you must do that from your perl script or whatever.

You also could try disabling the host lookup (use option -dhl) which also might spent lots of time in case of inappropriate network configuration.
Also, you Maybe should try out movescu manually for single studies to see how performance is and where exactly movescu _seems_ to hang (maybe that may also be a PACS problem?).

HTH, Regards,
Michael

[tchiang]

Thank you Michael for your response and suggestion.

I did try movescu manually on a single study to see its performance. That study contains only four images. It took more than 10 minutes to have the first Status=Pending message returned. It took more than 10 minutes on every Status=Pending for 3 or 4 times. It finally sent Status=Refused message but it already took more than an hour. I don't know the time (more than one hour) was spent on receiving the C-MOVE request or on actual sending images to the target DICOM device.

Thanks.

[Michael Onken]

Hi,

that is definetly not normal, except you are querying for really large images.

For that sounds like your network is not configured properly. Maybe the PACS has problems translating the receiver's AETitle to the movescu machine? Or are there any firewalls that may have impact on the DICOM network performance (try disabling all!). Also as suggested above, try option -dhl.

Another idea: If you are querying for compressed images (like "large" catheterization movies) and you only propose uncompressed transfer syntaxes, your PACS has to decompress all the images (with maybe dozens of frames each). If your PACS is not that fast, that may take a while, too. Which kind of data are you querying for?

Maybe it also makes sense that you post your movescu commandline call.

Regards,
Michael

[tchiang]

Thank you again for your response and suggestion, Michael.

Here is the command line fired up by my perl script:
/usr/local/dicom/bin/movescu -v -S -aem MYPACS01 -aec D250GEPACS -k 0008,0052=STUDY -k 0020,000D="1.2.840.113745.101000.1002000.38984.4641.21346839" 170.215.245.169 4100

My script is to pre-fetch ultrasound (US) mammogram priors (from a GE PACS to other DICOM devices). While most of the studies have less than 15 images, very few of them do have 300 to 500 or even more than 1000 images.

GE PACS requires a DAS (DICOM Application Server) to translate incoming/outgoing images to/from GE-proprietary format. I have a feeling that the load and performance of the GE PACD and DAS play a big role on the performance of my script and C-MOVE. On one of the tests which consists of 300 images, it only took around 25 minutes to return with Status=Refuse message as opposed of more than an hour for a study of only 4 images.

I will definitely try the option –dhl.

Thanks a lot.

[Michael Onken]

Hi,

sorry, I don't have any further ideas - your movescu call looks fine. So you want the PACS (AETitle D250GEPACS, IP 170.215.245.168, Port 4100) to send the study to another DICOM device (not movescu itself), with AETitle MYPACS01. What you (of course) must check is whether that MYPACS01 receiver is fully configured at the PACS, i. e. that the PACS can translate the AE Title MYPACS01 into the corresponding IP adress and port number.

Good luck,
Michael

[tchiang]

Thank you again, Michael. And yes, MYPACS01 receiver is fully configured at the PACS.

From my observation from the processing details, it seems that the bottleneck is the time wasted in waiting for Status=Pending. If the status is success, it transfers the study smooth and pretty quick. Therefore, I am considering using the following options. Since the manual page (from man movescu) contains limited description, I list the option I am interested in and my questions below:

--timeout 300 : Abandon the movescu process after waiting 5 minutes.
Question:
What are counted in these 300 seconds? Is it only the pending time or including processing time? For studies which need long processing time (for example studies with 500 or 1000 images), will this option terminate movescu in the middle of processing?

--cancel 3 : cancel after 3 responses
Question:
What responses are included in the 3 responses? I hope to abandon the movescu process after 3 Status=Pending responses. Is this the right option I should use?

Here is an example of processing details:
Sending query
Move SCU RQ: MsgID 1
Request:

# Dicom-Data-Set
# Used TransferSyntax: UnknownTransferSyntax
(0008,0052) CS [STUDY] # 6, 1 QueryRetrieveLevel
(0020,000d) UI [1.2.840.113745.101000.1002000.38853.9141.10708110] # 50, 1 StudyInstanceUID
Move Response 1: C-Move RSP: MsgID: 1 [Status=Pending]
AffectedSOPClassUID: =MOVEStudyRootQueryRetrieveInformationModel
Data Set: Not Present
NumberOfRemainingSubOperations: 6
NumberOfCompletedSubOperations: 0
NumberOfFailedSubOperations: 0
NumberOfWarningSubOperations: 0
Move Response 2: C-Move RSP: MsgID: 1 [Status=Pending]
AffectedSOPClassUID: =MOVEStudyRootQueryRetrieveInformationModel
Data Set: Not Present
NumberOfRemainingSubOperations: 6
NumberOfCompletedSubOperations: 0
NumberOfFailedSubOperations: 0
NumberOfWarningSubOperations: 0
Move Response 3: C-Move RSP: MsgID: 1 [Status=Pending]
AffectedSOPClassUID: =MOVEStudyRootQueryRetrieveInformationModel
Data Set: Not Present
NumberOfRemainingSubOperations: 6
NumberOfCompletedSubOperations: 0
NumberOfFailedSubOperations: 0
NumberOfWarningSubOperations: 0
Move Response 4: C-Move RSP: MsgID: 1 [Status=Pending]
AffectedSOPClassUID: =MOVEStudyRootQueryRetrieveInformationModel
Data Set: Not Present
NumberOfRemainingSubOperations: 5
NumberOfCompletedSubOperations: 0
NumberOfFailedSubOperations: 1
NumberOfWarningSubOperations: 0
Move Response 5: C-Move RSP: MsgID: 1 [Status=Pending]
AffectedSOPClassUID: =MOVEStudyRootQueryRetrieveInformationModel
Data Set: Not Present
NumberOfRemainingSubOperations: 4
NumberOfCompletedSubOperations: 0
NumberOfFailedSubOperations: 2
NumberOfWarningSubOperations: 0
Move Response 6: C-Move RSP: MsgID: 1 [Status=Pending]
AffectedSOPClassUID: =MOVEStudyRootQueryRetrieveInformationModel
Data Set: Not Present
NumberOfRemainingSubOperations: 1
NumberOfCompletedSubOperations: 0
NumberOfFailedSubOperations: 5
NumberOfWarningSubOperations: 0
C-Move RSP: MsgID: 1 [Status=Refused: OutOfResourcesSubOperations]
AffectedSOPClassUID: =MOVEStudyRootQueryRetrieveInformationModel
Data Set: Present
NumberOfCompletedSubOperations: 0
NumberOfFailedSubOperations: 6
NumberOfWarningSubOperations: 0Response Identifiers:

# Dicom-Data-Set
# Used TransferSyntax: BigEndianExplicit
(0008,0058) UI [1.2.840.113986.2.3567920692.20081015.130120.91\1.2.840.113986.2.3567920692.20081015.130217.502\1.2.840.113986.2.3567920692.20081015.130253.985\1.2.840.113986.2.3567920692.20081015.130326.764\1.2.840.113986.2.3567920692.20081015.130401.714\1.2.840.113986.2.3567920692.20081015.130437.751] # 286, 6 FailedSOPInstanceUIDList
Releasing Association
Requesting Association
Association Accepted (Max Send PDV: 28660)

Thanks a lot.

[Michael Onken]

Hi,

From my observation from the processing details, it seems that the bottleneck is the time wasted in waiting for Status=Pending.

Please note that all C-MOVE-RSP message are only very small messages sent to movescu about what is happening on the _second_ connection, ie. to let movescu know how many images were transferred to the target machine (which is _not_ movescu but your MYPACS01 machine). So it would be very unusual if movescu has a problem here. If you start the move request, movescu has nothing to do but waiting for intermediate C-MOVE-RSP messages (which are optional in DICOM after each image transferred on the second connection) and for the final C-MOVE-RSP message (required, sent after the last image was transferred successfully on the second connection). So movescu simply does nothing after start of image/study transfer but waiting for (any optional and one required) "progress status" messages. It's not very probable that this slows your transfer down.

If the status is success, it transfers the study smooth and pretty quick.

I think the communication between the PACS and the receiver of the images have some problems.

Therefore, I am considering using the following options. Since the manual page (from man movescu) contains limited description, I list the option I am interested in and my questions below:

--timeout 300 : Abandon the movescu process after waiting 5 minutes.
Question:
What are counted in these 300 seconds? Is it only the pending time or including processing time? For studies which need long processing time (for example studies with 500 or 1000 images), will this option terminate movescu in the middle of processing?

This timeout is only the TCP/IP timeout at the very beginning when movescu connects to your PACS. By the way, I guess none of the timeout option is of any help for your problem, sorry.

--cancel 3 : cancel after 3 responses
Question:
What responses are included in the 3 responses? I hope to abandon the movescu process after 3 Status=Pending responses. Is this the right option I should use?

As far as I can see after looking quickly into the code, this option makes movescu send a cancel message (meaning "please dont send any more images to the image receiver") to the PACS after receiving a certain amount of responses. Because responses for all images but the final images are optional, I guess this does only work for very specific use cases.

NumberOfRemainingSubOperations: 6
NumberOfCompletedSubOperations: 0
NumberOfFailedSubOperations: 0
NumberOfWarningSubOperations: 0

This is one of those optional intermediate responses, saying that there are 6 images on the second connection left to send.

Move Response 2: C-Move RSP: MsgID: 1 [Status=Pending]
AffectedSOPClassUID: =MOVEStudyRootQueryRetrieveInformationModel
Data Set: Not Present
NumberOfRemainingSubOperations: 6
NumberOfCompletedSubOperations: 0
NumberOfFailedSubOperations: 0
NumberOfWarningSubOperations: 0

This is the next one, still 6 images to send for the PACS on the second connection (a bit curious that the PACS sends intermediate messages without having done anyhting in between, but it's possible that this is a valid implementation of DICOM, I'm not sure).

Move Response 4: C-Move RSP: MsgID: 1 [Status=Pending]
AffectedSOPClassUID: =MOVEStudyRootQueryRetrieveInformationModel
Data Set: Not Present
NumberOfRemainingSubOperations: 5
NumberOfCompletedSubOperations: 0
NumberOfFailedSubOperations: 1
NumberOfWarningSubOperations: 0

Here you can already see, that one of the 6 images was sent by the PACS, _but_ the sub operation failed (NumberOfFailedSubOperations: 1. Sub operations always means the transfer on the connectio between PACS and image reciever), so the very first image couldnt be transferred.

Move Response 5: C-Move RSP: MsgID: 1 [Status=Pending]
AffectedSOPClassUID: =MOVEStudyRootQueryRetrieveInformationModel
Data Set: Not Present
NumberOfRemainingSubOperations: 4
NumberOfCompletedSubOperations: 0
NumberOfFailedSubOperations: 2
NumberOfWarningSubOperations: 0

2 images were tried to sent, but both failed...
-> and so on

C-Move RSP: MsgID: 1 [Status=Refused: OutOfResourcesSubOperations]
AffectedSOPClassUID: =MOVEStudyRootQueryRetrieveInformationModel
Data Set: Present
NumberOfCompletedSubOperations: 0
NumberOfFailedSubOperations: 6
NumberOfWarningSubOperations: 0Response Identifiers:

# Dicom-Data-Set
# Used TransferSyntax: BigEndianExplicit
(0008,0058) UI [1.2.840.113986.2.3567920692.20081015.130120.91\1.2.840.113986.2.3567920692.20081015.130217.502\1.2.840.113986.2.3567920692.20081015.130253.985\1.2.840.113986.2.3567920692.20081015.130326.764\1.2.840.113986.2.3567920692.20081015.130401.714\1.2.840.113986.2.3567920692.20081015.130437.751] # 286, 6 FailedSOPInstanceUIDList
Releasing Association
Requesting Association
Association Accepted (Max Send PDV: 28660)

This is the final one, stating that all 6 image transfers failed with overall status OutOfResourcesSubOperations, saying that on the second connection something went wrong. Also movescu is giving a list of SOP Insatance UIDs (6, one for every image) of those images the PACS said could not be transferred (all, in this case).

So, finally all information looks like the transfer between PACS and image receiver fails. movescu only protocols what the PACS tells per DICOM protocol about these failures but has nothing to do with the actual image transfer.

I guess the PACS is not able to convert the images to the transfer syntaxes needed for the image transfer on the second connection or the receiver just fails (maybe timeout) receiving the images. Maybe looking into your PACS logs would help you analyzing the problems.

Hope that helps...
Regards, Michael