Cuttlefish RADOS Bench 4M Object Throughput 4 Processes, 128 Concurrent Operations 2500 2000 BTRFS ) 1500 /s EXT4 B XFS t (M u p h g u ro 1000 h T 500 0 Write Read
Yeah, yeah, the bonded 10GbE network is maxed out. Good for you Mark.
Who cares about RADOS Bench though? I've moved to the cloud and do lots of small writes on block storage.
OK, if Ceph is so awesome why are you only testing 1 server? How does it scale?
Oak Ridge National Laboratory 4 Storage Servers, 8 Client Nodes DDN SFA10K Storage Chassis QDR Infiniband Everywhere A Boatload of Drives!
ORNL Multi-Server RADOS Bench Througput 4MB IOs, 8 Client Nodes 14000 12000 10000 Writes ) /s 8000 Reads B Writes (Including Journals) t (M u Disk Fabric Max p h g u 6000 Client Network Max ro h T 4000 2000 0 1 2 3 4 Server Nodes (11 OSDs Each)
So RADOS is scaling nicely. How much does data replication hurt us?
ORNL 4MB RADOS Bench Throughput 12000 10000 8000 ) /s B Write t (M u 6000 Read p h Total Write g u (Including Journals) ro h T 4000 2000 0 1 2 3 Replication Level
This is an HPC site. What about CephFS? NOTE: CephFS is not production ready! (Marketing and sales can now sleep again)
ORNL 4M CephFS (IOR) Throughput 7000 6000 5000 ) /s 4000 iB Max Write t (M Avg Write u p Max Read h g u 3000 Avg Read ro h T 2000 1000 0 1 2 3 4 5 6 7 8 Client Nodes (8 Processes Each)
Hundreds of Cluster Configurations Hundreds of Tunable Settings Hundreds of Potential IO Patterns Too Many Permutations to Test Everything!
When performance is bad, how do you diagnose?
Ceph Admin Socket
Blktrace & Seekwatcher
Where are we going from here?
More testing and Bug fixes! Erasure Coding Cloning from Journal Writes (BTRFS) RSOCKETS/RDMA Tiering