Slow transfer rates

All other questions regarding DCMTK

Moderator: Moderator Team

Post Reply
Message
Author
dimitri
Posts: 39
Joined: Mon, 2012-04-23, 13:43

Slow transfer rates

#1 Post by dimitri »

Dear DCMTK Forum members,

We have been experiencing slow transfer rates from our Siemens MRI consoles to storescp (DCMTK 3.6.6).

After reading "FAQ #48: Slow transfer when sending many small DICOM images", I tried the following suggestion:

Code: Select all

export TCP_NODELAY=1
It seems to be working very well. Thank you DICOM@OFFIS for DCMTK and the nice FAQ.

I am just wondering about this sentence:
Starting with DCMTK 3.6.2, the Nagle algorithm is not disabled by default since this does not seem to be appropriate (anymore) for most modern operating systems.
What do you mean by "modern operating systems"? Which operating system is relevant? The OS of the MRI consoles? The OS of the server running storescp?

In our case:
  • the server runs CentOS 6.5 (yes, I know, we need to update it :)),
  • the consoles of the 3 T Siemens Prisma Fit run syngo MR E11 on Windows 7,
  • the consoles of the 7 T Siemens MRI run syngo MR B17 on Windows XP.
So far, we have tested data transfer from the 3 T consoles running Windows 7 and have seen tremendous improvement with TCP_NODELAY set. Would you qualify CentOS 6.5 and Windows 7 as OSes that fall outside of the "modern operating systems" category? Is there anything else we should look into?

J. Riesmeier
DCMTK Developer
Posts: 2501
Joined: Tue, 2011-05-03, 14:38
Location: Oldenburg, Germany
Contact:

Re: Slow transfer rates

#2 Post by J. Riesmeier »

What do you mean by "modern operating systems"? Which operating system is relevant? The OS of the MRI consoles? The OS of the server running storescp?
The relevant OS is the one a DCMTK-based network tool is running on. Unfortunately, there is no simple answer to the question whether the Nagle algorithm should be disabled or not. Because of feedback we've received from DCMTK users and based on our own tests, we've decided to change the default behavior some years ago.

dimitri
Posts: 39
Joined: Mon, 2012-04-23, 13:43

Re: Slow transfer rates

#3 Post by dimitri »

A colleague experienced similar performance issues when running DCMTK on virtual CentOS 8 servers, and again performance has reportedly improved by disabling the Nagle algorithm. I keep wondering why we need to disable the Nagle algorithm on modern operating systems.
Last edited by dimitri on Tue, 2022-05-17, 14:39, edited 1 time in total.

dimitri
Posts: 39
Joined: Mon, 2012-04-23, 13:43

Re: Slow transfer rates

#4 Post by dimitri »

After disabling the Nagle algorithm with TCP_NODELAY, operators report the following performance improvements:
* Before setting TCP_NODELAY, the estimated time to send 1.62 GB / 6028 files from a console of our 7 T Siemens MRI running syngo MR B17 on Windows XP to our server running storescp was more than 10 minutes. We would also experience frequent console freezes, reportedly sometimes requiring a reboot of the console to resume data transfer.
* After setting TCP_NODELAY, operators measured the time to send the same dataset from the same console to the same server to be only 2 minutes 15 seconds. And no more console freezes.
* Sending the same dataset from a 7 T console to a 3 T console takes 4 minutes 37 seconds.

So at least, after disabling TCP_NODELAY, we have been doing better than Siemens when sending data between their consoles.

However, I remain puzzled: 2 minutes 15 seconds to send 1.62 GB means a transfer rate of 6.5 MB/s, approximately 50 Mb/s, which remains rather low. Our servers and network equipments support transfer rates of 1 Gb/s. In your experience, is this related to the many small DICOM files, their indexing in the console database and the overhead to process each individual file? Is this something that could be improved, perhaps by consulting OFFIS?
Last edited by dimitri on Tue, 2022-05-17, 18:15, edited 2 times in total.

Michael Onken
DCMTK Developer
Posts: 2048
Joined: Fri, 2004-11-05, 13:47
Location: Oldenburg, Germany
Contact:

Re: Slow transfer rates

#5 Post by Michael Onken »

Hi,

DICOM does not have that much overhead in the network protocol, just a few byte per PDU package (16KB default in most DCMTK tools).

You could try to increase the maximum DICOM PDU size that the sender is allowed to use by using storescp's option -pdu:

Code: Select all

-pdu  --max-pdu              [n]umber of bytes: integer (4096..131072)
                             set max receive pdu to n bytes (default: 16384)
and set it to something bigger than the default 16K. Though I don't think this will bring a huge speedup unless TCP/IP windowing sizes or the Siemens implementation are using unfavorable settings related to the default PDU size.
Also, if you have a delay in the very beginning of sending (up to 30 seconds), use option -dhl to disable DNS lookup within storescp.

If this does not help, try to send images using one of the OFFIS tools, e.g. storescu or dcmsend to see how this combination performs on the same network. Then try tools from other toolkits to find out whether this is a DCMTK issue.

Sometimes also sniffing helps on the network (if you are allowed to) to see whether it is under high load by other systems or whether there are lots of collisions and related retransmissions on the TCP/IP level.

Best regards,
Michael

Marco Eichelberg
OFFIS DICOM Team
OFFIS DICOM Team
Posts: 1437
Joined: Tue, 2004-11-02, 17:22
Location: Oldenburg, Germany
Contact:

Re: Slow transfer rates

#6 Post by Marco Eichelberg »

Another factor that limits the network throughput is the serial mode of operation:
  • modality sends a single image (slice)
  • receiver stored that image (perhaps including database operations)
  • receiver send acknowledgement (C-STORE Response) back to modality
  • modality sends second image...
The DICOM network protocol has an asynchronous mode that avoids this problem, but unfortunately nobody (including DCMTK) supports that.

In the storescp tool, there is a command line option called --bit-preserving that you might want to try. It directly stores incoming images on the hard disk without buffering them in memory and feeding them through the DICOM parser. In many cases this is significantly faster because the acknowledgement can be sent directly after receipt of the last packet. Some storescp command line options won't work in that mode (e.g. you cannot store the files in a different transfer syntax than the one used over the network connection), but that should not be much of an issue.

dimitri
Posts: 39
Joined: Mon, 2012-04-23, 13:43

Re: Slow transfer rates

#7 Post by dimitri »

I'm not so certain any more that TCP_NODELAY saved the day. That's what operators had told me, but then I've compared the figures they had sent back in February (without TCP_NODELAY) and in May (with TCP_NODELAY), and they seem similar to me, not to say that it's worse with TCP_NODELAY :(

I intend to "close" this thread and open a new one with accurate measurement:
- using my laptop to send the same dataset with storescu and gdcmscu,
- starting in the VLAN of the server, and then moving the laptop to the VLAN of the MRI consoles if necessary, going as far as using the same RJ45 jack as an MRI console to be on the safe side,
- testing and comparing your suggestions other than TCP_NODELAY: -pdu, -dhl, --bit-preserving, ...
- comparing transfer rates with DICOM and SSH,
- ...

is there a way to instruct storescp to not write on disk, to avoid disk I/O issues?

Michael Onken
DCMTK Developer
Posts: 2048
Joined: Fri, 2004-11-05, 13:47
Location: Oldenburg, Germany
Contact:

Re: Slow transfer rates

#8 Post by Michael Onken »

Hi,

Try option --ignore.

Best regards,
Michael

dimitri
Posts: 39
Joined: Mon, 2012-04-23, 13:43

Re: Slow transfer rates

#9 Post by dimitri »

Following up with your suggestions in thread Slow transfer rates (this time with actual measurements).

Post Reply

Who is online

Users browsing this forum: Ahrefs [Bot], Bing [Bot], Google [Bot] and 1 guest