Improving RD5 file reading performance in with high latency storage

[scarapella]

Hello,

I've been deploying a dockerized brouter using cloud storage buckets and because of the way the RD5 files are read, the performance is... less than idea. By introducing this optional buffered reader implementation it significantly improves the performance.

Buffered reader is switched on using -DuseBufferedReader=true the sized of the read buffer can optionally be adjusted using -DreadBufferSize=2m. Although the default of 1M seems pretty good compromise.

-scarapella

[afischerdev]

@scarapella
Thank you for your new ideas and their implementation.
However, I can begin testing after the migration to a new server. This will take some time, though.

[afischerdev]

@scarapella

the performance is... less than idea.

Please let us know: How did you measure that?

On my tests with the jar from this PR I did around 160 routing calls to the server with result sizes from 1k to 600k.
This takes on a local test around 26-27 seconds With both variants -DuseBufferedReader on and off.
Same results with a run inside docker - 32-33 seconds.

I can't see a difference.

[scarapella]

I assume you use local storage for your RD5 files? In that case I agree i would not expect a big difference since I think the issue is endemic to the setup I am using where the storage latency is much higher (using cloud bucket storage) I'll setup a side by side comparison soon as I'm able to benchmark some differences.

[scarapella]

@afischerdev do you have a preferred test set?

[afischerdev]

@scarapella
Great, thanks for the offer - this are my favorites
E0_N40.rd5
E0_N45.rd5
E0_N50.rd5
E0_N55.rd5
E10_N45.rd5
E10_N50.rd5
E10_N55.rd5
E15_N45.rd5
E20_N50.rd5
E25_N55.rd5
E25_N70.rd5
E35_N20.rd5
E35_N25.rd5
E35_N30.rd5
E40_N20.rd5
E40_N25.rd5
E45_N15.rd5
E45_N20.rd5
E45_N25.rd5
E5_N45.rd5
E5_N50.rd5
E5_N55.rd5
W30_N35.rd5
W5_N40.rd5
W5_N45.rd5
W5_N50.rd5