Context Navigation

← Previous Ticket
Next Ticket →

Ticket #11614 (closed: fixed)

Opened 5 years ago

Last modified 5 years ago

SaveNXSPE: Write speeds very slow on ceph file system

Reported by:	Martyn Gigg	Owned by:	Martyn Gigg
Priority:	critical	Milestone:	Release 3.4
Component:	Framework	Keywords:
Cc:		Blocked By:
Blocking:		Tester:	Andrei Savici

Description

The excitations group have noticed that a 300Mb file can take ~20mins to write to the ceph file system used for autoreduction!

Change History

comment:1 Changed 5 years ago by Martyn Gigg

The following script uses h5py to verify the problem:

import h5py
import numpy as np
import os
import time
import sys

def write_data(data_group, name, data, slab_size):
    """
    slab_size in bytes
    """
    dset = data_group.create_dataset(name, shape=data.shape,
                                     dtype='f')
    nrows, ncols = data.shape
    nrows_block = slab_size/(8*ncols)
    if nrows_block == 0:
        # not enough for whole slab
        nrows_block = nrows

    for row_idx in range(0, nrows-nrows_block, nrows_block):
        dset[row_idx:row_idx + nrows_block,:] = data[row_idx:row_idx + nrows_block,:]

    # final block
    remainder = nrows % nrows_block
    if remainder > 0:
        start, stop = nrows - remainder, nrows+1
        dset[start:stop,:] = data[start:stop,:]

def write_file(filename, signal, errors, slab_size):
    try:
        os.remove(filename)
    except OSError:
        pass
    hdf_file = h5py.File(filename, 'w')

    # Top level group
    root_group = hdf_file.create_group("workspace_1")
    root_group.attrs["NX_class"] = "NXentry"

    # data group
    data_group = root_group.create_group("data")
    data_group.attrs["NX_class"] = "NXdata"

    # write the data
    write_data(data_group, "data", signal, slab_size)
    write_data(data_group, "error", errors, slab_size)

    hdf_file.close()

#------------------------------------------------------

if len(sys.argv) < 2:
    print "Usage: %s FILEPATH" % sys.argv[0]
    sys.exit(1)

# Gives ~ 300Mb dataset
num_rows = 286720
num_cols = 66
signal = np.arange(num_rows*num_cols)
signal = signal.reshape(num_rows, num_cols)
errors = np.sqrt(signal)

# 4Mb
slab_size = 4*1024*1024
# 1 row (bytes)
#slab_size = 8*num_cols

start_time = time.time()
write_file(sys.argv[1], signal, errors, slab_size)
end_time = time.time()

print "Time to write file: {0}s".format(end_time - start_time)

If you uncomment the slab_size = 8*num_cols line then you will observe a large difference in the write speed. This is because more calls to putslab mean more disk accesses and some parts of ceph must be waiting on previous chunks to be transferred to remote locations before allowing the next to be written.

A solution is to write the data in larger chunks.

Last edited 5 years ago by Martyn Gigg (previous) (diff)