Slow transfer rates (this time with actual measurements)

All other questions regarding DCMTK

Moderator: Moderator Team

Message
Author
dimitri
Posts: 39
Joined: Mon, 2012-04-23, 13:43

Slow transfer rates (this time with actual measurements)

#1 Post by dimitri »

I am restarting the Slow transfer rates thread, this time including measurements of transfer rates between a Linux Ubuntu client machine and the CentOS 6.5 server running storescp (initially DCMTK 3.6.6 and now DCMTK 3.6.7).

I will be sending two series of DICOM files:
  • 000017_mbep2d-iPAT3-MB2-1pt2mm-AP-run2
    • number of files: 399
    • total size: 1.6 GB
  • 000021_mp2rage-iso0.65-iPAT2-angulated-UNI-Images
    • number of files: 240
    • total size: 66 MB
A typical attempt to send the above series with storescu outside office hours, with little network traffic, looks like this:

Code: Select all

$ time storescu neurospin-dicom.xxxxx.xxx.xx 1104 --scan-directories 000017_mbep2d-iPAT3-MB2-1pt2mm-AP-run2

real	0m42,180s
user	0m1,210s
sys	0m1,890s
$ 
$ time storescu neurospin-dicom.xxxxx.xxx.xx 1104 --scan-directories 000021_mp2rage-iso0.65-iPAT2-angulated-UNI-Images

real	0m7,526s
user	0m0,280s
sys	0m0,229s
$ 
The transfer rate for the largest set reaches 39 MB/s (~ 300 Mb/s), a bit slower than an SSH transfer:

Code: Select all

$ time scp -qr 000017_mbep2d-iPAT3-MB2-1pt2mm-AP-run2 neurospin-dicom.xxxxx.xxx.xx:/tmp

real	0m31,374s
user	0m5,206s
sys	0m5,300s
$ 
$ time scp -qr 000021_mp2rage-iso0.65-iPAT2-angulated-UNI-Images neurospin-dicom.xxxxx.xxx.xx:/tmp

real	0m1,747s
user	0m0,312s
sys	0m0,315s
$ 
Last edited by dimitri on Mon, 2022-05-30, 13:58, edited 3 times in total.

dimitri
Posts: 39
Joined: Mon, 2012-04-23, 13:43

Re: Slow transfer rates (this time with actual measurements)

#2 Post by dimitri »

Any way, sending data from my Linux machine is indeed faster than from the Siemens consoles. I still plan on testing some of the storescp options suggested in the previous thread.(--bit-preserving, --max-pdu, --ignore, TCP_NODELAY) when executing C-STORE operations from my Linux machine, just in case.

Then I would like to test a different tool to send the two series. Unfortunately gdcmscu does not seem to be working on my Linux machine, unless I am doing something wrong:

Code: Select all

$ gdcmscu --store neurospin-dicom.xxxxx.xxx.xx 1104 --recursive --input 000017_mbep2d-iPAT3-MB2-1pt2mm-AP-run2
gdcmscu: /build/gdcm-NQLsIX/gdcm-2.8.4/Applications/Cxx/gdcmscu.cxx:312: int main(int, char**): Assertion `0' failed.
Abandon (core dumped)
$ 
Can you perhaps suggest an alternative command I could run from the Linux command line to test sending the two series with C-STORE operations?
Last edited by dimitri on Tue, 2022-05-24, 06:55, edited 1 time in total.

dimitri
Posts: 39
Joined: Mon, 2012-04-23, 13:43

Re: Slow transfer rates (this time with actual measurements)

#3 Post by dimitri »

Oh, it looks like gdcmscu doesn't like the documented long --input option, I have to use the short -i option instead. With that bug out of the way, I get similar transfer rates, with gdcmscu actually a bit slower on larger data:

Code: Select all

$ time gdcmscu --store neurospin-dicom.xxxxx.xxx.xx 1104 --recursive -i 000017_mbep2d-iPAT3-MB2-1pt2mm-AP-run2

real	0m50,368s
user	0m4,379s
sys	0m6,908s
$ 
$ time gdcmscu --store neurospin-dicom.xxxxx.xxx.xx 1104 --recursive -i 000021_mp2rage-iso0.65-iPAT2-angulated-UNI-Images

real	0m7,247s
user	0m0,341s
sys	0m0,280s
$ 

dimitri
Posts: 39
Joined: Mon, 2012-04-23, 13:43

Re: Slow transfer rates (this time with actual measurements)

#4 Post by dimitri »

Option --ignore does have a radical effect. Data transfer speeds now exceed SSH speeds, typically:

Code: Select all

$ time storescu neurospin-dicom.xxxxx.xxx.xx 1104 --scan-directories 000017_mbep2d-iPAT3-MB2-1pt2mm-AP-run2

real	0m21.488s
user	0m1.300s
sys	0m1.779s
$ 
$ time storescu neurospin-dicom.xxxxx.xxx.xx 1104 --scan-directories 000021_mp2rage-iso0.65-iPAT2-angulated-UNI-Images

real	0m4.327s
user	0m0.309s
sys	0m0.180s
$ 
That's 77 MB/s, almost 600 Mb/s. It makes sense since:
  • DICOM does not encrypt the data stream so storescp has the potential to be faster than SSH,
  • without --ignore, storescp writes data onto an NFS share, which is of course much slower than local disk, especially with the pathologically large number of small files that characterizes DICOM datasets.
Of course at this point the question remains why transfer speeds from Siemens consoles are slower.

dimitri
Posts: 39
Joined: Mon, 2012-04-23, 13:43

Re: Slow transfer rates (this time with actual measurements)

#5 Post by dimitri »

Option --bit-preserving does not help further improve transfer rates from the Linux machine. I guess it won't help much with Siemens consoles either. Plus it is not compatible with --sort-on-patientname which we use with --eostudy-timeout.

Option -dhl does not seem to be helping either in this case.

dimitri
Posts: 39
Joined: Mon, 2012-04-23, 13:43

Re: Slow transfer rates (this time with actual measurements)

#6 Post by dimitri »

Finally I have tested setting TCP_NODELAY again and I do see better transfer rates, together with --ignore 104 MB/s or approximatively 830 Mb/s for the large dataset and an impressive × 4 improvement for the small dataset (although the transfer speed remains much lower due to the startup overhead I guess):

Code: Select all

$ time storescu neurospin-dicom.xxxxx.xxx.xx 1104 --scan-directories 000017_mbep2d-iPAT3-MB2-1pt2mm-AP-run2

real	0m15.727s
user	0m1.247s
sys	0m1.746s
$ 
$ time storescu neurospin-dicom.xxxxx.xxx.xx 1104 --scan-directories 000021_mp2rage-iso0.65-iPAT2-angulated-UNI-Images

real	0m1.116s
user	0m0.359s
sys	0m0.082s
$ 
At some point we will move storescp services to a more recent CentOS 8 or Ubuntu 22.04 server. We will then be able to investigate whether TCP_NODELAY has the same effect on more recent systems.

By the way, I see no difference when testing during peak office hours. It makes sense because I know our local network is seldom saturated.
Last edited by dimitri on Mon, 2022-05-30, 11:51, edited 2 times in total.

Michael Onken
DCMTK Developer
Posts: 2048
Joined: Fri, 2004-11-05, 13:47
Location: Oldenburg, Germany
Contact:

Re: Slow transfer rates (this time with actual measurements)

#7 Post by Michael Onken »

Hi,

thank you for the interesting measurements which also may help other people. Did you also try using different PDU sizes (try some values on storescp side, e.g. "-pdu 100000"), maybe this could also speed up the larger datasets.

Best regards,
Michael

dimitri
Posts: 39
Joined: Mon, 2012-04-23, 13:43

Re: Slow transfer rates (this time with actual measurements)

#8 Post by dimitri »

I have just tried -pdu 100000, in conjunction with TCP_NODELAY=1 or not, it has no visible effect.

The next step would be to test these same storescp options when sending data from the Siemens consoles. Unfortunately I need to be in front of the MRI consoles to send data, I cannot do that remotely, so it will take some time.

Michael Onken
DCMTK Developer
Posts: 2048
Joined: Fri, 2004-11-05, 13:47
Location: Oldenburg, Germany
Contact:

Re: Slow transfer rates (this time with actual measurements)

#9 Post by Michael Onken »

I have just tried -pdu 100000, in conjunction with TCP_NODELAY=1 or not, it has no visible effect.
Ok, probably Siemens uses a hard-configured DICOM packet size, no matter what the SCP offers.

dimitri
Posts: 39
Joined: Mon, 2012-04-23, 13:43

Re: Slow transfer rates (this time with actual measurements)

#10 Post by dimitri »

Michael Onken wrote: Mon, 2022-05-30, 14:34 Ok, probably Siemens uses a hard-configured DICOM packet size, no matter what the SCP offers.
I plan on testing against the Siemens MRI consoles in the next few weeks. Would you be able to recommend a simple way to identify this hard-configured DICOM packet size? Just try different -pdu arguments?

Michael Onken
DCMTK Developer
Posts: 2048
Joined: Fri, 2004-11-05, 13:47
Location: Oldenburg, Germany
Contact:

Re: Slow transfer rates (this time with actual measurements)

#11 Post by Michael Onken »

dimitri wrote: Thu, 2022-06-09, 20:07
Michael Onken wrote: Mon, 2022-05-30, 14:34 Ok, probably Siemens uses a hard-configured DICOM packet size, no matter what the SCP offers.
I plan on testing against the Siemens MRI consoles in the next few weeks. Would you be able to recommend a simple way to identify this hard-configured DICOM packet size? Just try different -pdu arguments?
The PDU size is communicated during association negotiation. Use any DCMTK network tool with --debug (-d) to see the PDU size in the log.

Best regards,
Michael

dimitri
Posts: 39
Joined: Mon, 2012-04-23, 13:43

Re: Slow transfer rates (this time with actual measurements)

#12 Post by dimitri »

I am not certain I can install DCMTK on the Siemens MRI consoles. In any case, between my Linux machine and the Linux storescp server, currently running DCMTK 3.6.7:

Code: Select all

$ storescu --debug neurospin-dicom.xxxxx.xxx.xx 1104 --scan-directories 000021_mp2rage-iso0.65-iPAT2-angulated-UNI-Images
[...]
D: ====================== BEGIN A-ASSOCIATE-AC =====================
D: Our Implementation Class UID:      1.2.276.0.7230010.3.0.3.6.2
D: Our Implementation Version Name:   OFFIS_DCMTK_362
D: Their Implementation Class UID:    1.2.276.0.7230010.3.0.3.6.7
D: Their Implementation Version Name: OFFIS_DCMTK_367
D: Application Context Name:    1.2.840.10008.3.1.1.1
D: Calling Application Name:    STORESCU
D: Called Application Name:     ANY-SCP
D: Responding Application Name: ANY-SCP
D: Our Max PDU Receive Size:    16384
D: Their Max PDU Receive Size:  16384
[...]

Michael Onken
DCMTK Developer
Posts: 2048
Joined: Fri, 2004-11-05, 13:47
Location: Oldenburg, Germany
Contact:

Re: Slow transfer rates (this time with actual measurements)

#13 Post by Michael Onken »

Hi,

yes, that's exactly the right place to look at:
D: Our Max PDU Receive Size: 16384
D: Their Max PDU Receive Size: 16384
This means both systems accept maximally 16K (DCMTK default) as maximum PDU size when the other system sends DICOM data.

Best,
Michael

dimitri
Posts: 39
Joined: Mon, 2012-04-23, 13:43

Re: Slow transfer rates (this time with actual measurements)

#14 Post by dimitri »

I have started testing C-STORE requests, from the console of our 7 T Siemens MRI running syngo MR B17 on Windows XP, to our test storescp instance.

I started with our test storescp logs set to DEBUG, which shows the Max PDU size sent by the Siemens console is 262144:

Code: Select all

[...]
2022-06-22 09:54:16.964 DEBUG: DUL: disabling Nagle algorithm as requested at runtime (TCP_NODELAY=1)
[...]
2022-06-22 09:54:16.972  INFO: Association Received
2022-06-22 09:54:16.973 DEBUG: Parameters:
2022-06-22 09:54:16.973 DEBUG: ====================== BEGIN A-ASSOCIATE-RQ =====================
2022-06-22 09:54:16.973 DEBUG: Our Implementation Class UID:      1.2.276.0.7230010.3.0.3.6.7
2022-06-22 09:54:16.973 DEBUG: Our Implementation Version Name:   OFFIS_DCMTK_367
2022-06-22 09:54:16.973 DEBUG: Their Implementation Class UID:    1.3.12.2.1107.5.2
2022-06-22 09:54:16.973 DEBUG: Their Implementation Version Name: MR_VB17A
2022-06-22 09:54:16.973 DEBUG: Application Context Name:    1.2.840.10008.3.1.1.1
2022-06-22 09:54:16.973 DEBUG: Calling Application Name:    MRC18917
2022-06-22 09:54:16.973 DEBUG: Called Application Name:     PtkDSU-NS-TEST
2022-06-22 09:54:16.973 DEBUG: Responding Application Name: 
2022-06-22 09:54:16.973 DEBUG: Our Max PDU Receive Size:    16384
2022-06-22 09:54:16.973 DEBUG: Their Max PDU Receive Size:  262144
2022-06-22 09:54:16.973 DEBUG: Presentation Contexts:
[...]
Last edited by dimitri on Wed, 2022-06-22, 09:23, edited 1 time in total.

Michael Onken
DCMTK Developer
Posts: 2048
Joined: Fri, 2004-11-05, 13:47
Location: Oldenburg, Germany
Contact:

Re: Slow transfer rates (this time with actual measurements)

#15 Post by Michael Onken »

You should then try to experiment with a different PDU size on storescp, since storescp tells the other party which PDU size they can maximally send to storescp (to see whether it makes a difference)

Post Reply

Who is online

Users browsing this forum: Ahrefs [Bot], Bing [Bot], Google [Bot] and 1 guest