Slow storescu/scp transfer

[budric]

Hi,

I find DICOM transfer to be very slow. I setup a simple test to measure performance:

Server

Code: Select all

storescp -v --implicit --max-pdu 131072 5678

Client

Code: Select all

time storescu --scan-directories localhost 5678 0000004/

The result of time was:

Code: Select all

real    2m57.487s
user    0m0.735s
sys     0m0.053s

The dataset size is about 120 MB, so the transfer rate is 0.66 MB/second. This is very slow and I can assure you the disk io, cpu is capable of much better. Also it's running on localhost so there should be very minimal network delay for all the ACK packets. bmon shows lo interface at an average of 650 KiB/s. PDU is set to maximum and also tried default 16k.

I tried versions v3.6.0 2011-01-06 and v3.6.1 2012-11-02. The 3.6 is from ubuntu apt repository, and I compiled 3.6.1. They should not be debug binaries.

Any advice on what I'm doing wrong? Can anyone confirm same performance, better, worse performance on their system?

Thanks.

[J. Riesmeier]

I cannot confirm this bad performance. I just transferred 131 DICOM images with a total size of 1.2 GB in ~3 seconds. Without storing the received datasets as files (storescp option --ignore), it took ~2 seconds.

[st80rules]

I have the same performance problem. On the same network, transferring DICOM images to/from an Osirix station is more than twice as fast as transferring to/from another machine that uses our software (and dcmtk). Is there any tweaking that can be done to accelerate image transfers?

[budric]

This is very strange. I'm not getting anywhere near those transfer rates. More server information:
Linux Kernel: 3.11.0-14-generic

bandwidth test:

Code: Select all

iperf -c localhost
------------------------------------------------------------
Client connecting to localhost, TCP port 5001
TCP window size:  648 KByte (default)
------------------------------------------------------------
[  3] local 127.0.0.1 port 54572 connected with 127.0.0.1 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  57.8 GBytes  49.7 Gbits/sec

disk io :

Code: Select all

dd bs=1M count=256 if=/dev/zero of=test conv=fdatasync
256+0 records in
256+0 records out
268435456 bytes (268 MB) copied, 0.723655 s, 371 MB/s

My compile procedure (trying a newer branch as well)

Code: Select all

git clone git://git.dcmtk.org/dcmtk.git dcmtk
cd dcmtk
git checkout DCMTK-3.6.1_20131114
mkdir build && cd build
cmake -DCMAKE_BUILD_TYPE=Release ../
make -j

[J. Riesmeier]

First of all, you should make sure that the hostname lookup does not cause timeout issues. The various SCP tools in the DCMTK have an -dhl option for this purpose. Also playing around with the PDU size might help. However, I have no idea why OsiriX should be twice as fast (since it is probably also using DCMTK for networking).

[J. Riesmeier]

Ok, I again performed the tests with 131 files (see above). The first call is slower but still around 9 seconds for reading via network (1 GBit/s NAS with hard disks), transferring to localhost and storing as files (on a SSD). When I repeat the same call, I get something around 5 seconds. The third call is below 5 seconds. Using --ignore (i.e. not storing of files on the receiver side) and having the files (apparently) in the cache, I get something between 2 and 3 seconds.

Don't ask me why the results were differently this morning.

Of course, I'm using a recent Linux system

[budric]

I tried it on Windows 7 and it does indeed work on the order of 10 seconds (I don't have time utility on windows). Very bizarre. Same arguments. May I ask what system you were testing on?

Here's some more testing I did:

I converted the dataset to be little endian explicit

Code: Select all

find . -type f -print0 | xargs -0 -I file dcmconv --write-xfer-little file file

I ran storescu with debug enabled, it doesn't appear to be converting transfer syntax. Sample debug output

Code: Select all

I: Sending file: /tmp/test_dcm/<snip>
D: DcmItem::checkTransferSyntax() TransferSyntax="Little Endian Explicit"
I: Transfer Syntax: LittleEndianExplicit -> LittleEndianExplicit
I: Sending Store Request: MsgID 170, (CT)
D: ===================== OUTGOING DIMSE MESSAGE ====================
D: Message Type                  : C-STORE RQ
D: Message ID                    : 170
D: Affected SOP Class UID        : CTImageStorage
D: Affected SOP Instance UID     : <snip>
D: Data Set                      : present
D: Priority                      : low
D: ======================= END DIMSE MESSAGE =======================
I: Received Store Response
D: ===================== INCOMING DIMSE MESSAGE ====================
D: Message Type                  : C-STORE RSP
D: Presentation Context ID       : 41
D: Message ID Being Responded To : 170
D: Affected SOP Class UID        : CTImageStorage
D: Affected SOP Instance UID     : <snip>
D: Data Set                      : none
D: DIMSE Status                  : 0x0000: Success
D: ======================= END DIMSE MESSAGE =======================

PDU set to 16k does appear to be faster on this system. I got it down to 100 seconds (from 180).

I don't think it's disk read issue since this executes in no time on the folder full of dicom.

Code: Select all

time find . -type f -print0 | xargs -0 -I file cat file > /dev/null

[J. Riesmeier]

May I ask what system you were testing on?

Code: Select all

$ uname -a
Linux thinkpad 3.5.0-40-generic #62-Ubuntu SMP Thu Aug 22 00:55:19 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux

The main memory is 8 GB and here is the CPU info:

Code: Select all

$ lscpu
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                4
On-line CPU(s) list:   0-3
Thread(s) per core:    2
Core(s) per socket:    2
Socket(s):             1
NUMA node(s):          1
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 58
Stepping:              9
CPU MHz:               1200.000
BogoMIPS:              5187.86
Virtualization:        VT-x
L1d cache:             32K
L1i cache:             32K
L2 cache:              256K
L3 cache:              3072K
NUMA node0 CPU(s):     0-3

As I said, the local drive is a fast SSD (used for saving the received files) and the network drive is a 1 GBit/s connected conventional harddisk in a NAS (used for reading the files to be transmitted).