Move workspace file based operations to ISaveable class and simplify file-based operations

[Alex Buts]

Current structure of fileBased event workspace is very complex. The file based/ not-file based operations are controlled by five boolean parameters, and user has to set all of them in correct order to get working file-based algorithm.

The Kernel::DiskBuffer tests introduces some algorithm to work with file based workspaces and these algorithms are repeated within different MDAlgorithms.

There are two independent modes of working with file based workspaces and it is unclear which one to use.

The purpose of this ticket is to simplify the construction of file-based MDWorkspaces, move main file-related functionality into ISaveAble DiskBuffer couple of objects and simplify MDBoxes/MDGridBoxes removing this functionality from them.

[Alex Buts]

refs #6362 Attempt to move the code responsible to ISaveable

into Kernel

Changeset: a85e99144b3ef5ac3885af6699c408fe0e00fbf1

[Alex Buts]

refs #6362 Fixing ISavable

Changeset: 48e61e6d43fee52433d1d0d40256bb547b11aaa7

[Alex Buts]

refs #6362 ISaveable modified to keep the information about its place

ISavable now knows about its position in file and works together with the DiskBuffer to identify the location of the data in the file.

Changeset: b385e85cb27721828a83f7c4b3412e6c6b512341

[Alex Buts]

refs #6362 Checked performance tests

Changeset: 328e6c9e76431b0e467061b670125a6865b58d5a

[Alex Buts]

refs #6362 MDGridBox test works

(without modifications)

Changeset: 9e5bd15f20b9d39d813547851c0be0a00201dfb1

[Alex Buts]

refs #6362 Save made constant and special command is used to free mem

small changes to algorithms too to incorporate new logic of working

Changeset: 8b510312a3f58d2b9449996bcfd07c822f11be00

[Alex Buts]

refs #6362 SaveMD works

Changeset: 65686144415f72ebb040af9a52288baefc011c21

[Alex Buts]

refs #6362 Removed notuse disk buffer mode from MDWorkspace modes

All file based workspaces have to use disk buffer now. If the disk buffer size is set to 0, the data are saved to HDD (or discarded if not modified) as soon as they are free.

Changeset: 0d8a0780b9fe6826e6ee986e37d2dcd93ba25ee2

[Alex Buts]

refs #6362 LoadMD tests work

Changeset: 8ef22cf32374cc790ccda39a9c6a750569a0a726

[Alex Buts]

refs #6362 SliceMD reconfigured to use write output buffer

Changeset: 8704af4e3d0b9080c74d993bfb3632c7eb374b21

[Alex Buts]

refs #6362 Modified MDBoxBaseTest to reflect new work patterns

Changeset: 0e68c86d3752226da3bd435f97cf75ed1fd62411

[Alex Buts]

refs #6362 UseWrieBuffer meth is available on BoxController interface

and always reports true for export to python.

Changeset: 0dccf3bc49874561175ac72a4d54d4e3e9b549a5

[Alex Buts]

refs #6362 Commented yet unused file from cmake list

Changeset: 7811e0ecb200362ca7b4feab7a8af60a9df270f0

[Alex Buts]

refs #6362 fixing Unix error

Changeset: fe217c03a8242511f5205abd1a88ef9c81b9b183

[Alex Buts]

refs #6362 fixing Unix error (missing boost header)

Changeset: 7bc34073e41879dc4927cf903fe08b6f5d6c1f5e

[Alex Buts]

refs #6362 attempt to fix mutexing issue on Unix

(not fully tested yet on Win but there are hopes. )

Changeset: ed3b4a2f56531b4121a0968403663d492f69ba52

[Alex Buts]

refs #6362 Attempt to use recursive mutex in suspicious places only

Changeset: 83beea3c2ecb887d137302a9b5e21e5047ecffcb

[Alex Buts]

refs #6362 SaveMDTest deletes files before and after testing

(this is to fix recent problems with tests)

Changeset: 88564dc671ca635d8d8a94f88612b811f4fc69fa

[Alex Buts]

refs #6362 All failing tests trying to delete files before and after

Changeset: 78bd02d2a840ba8e929e333557ef40a4bb2aeec0

[Alex Buts]

refs #6362 Now it will hopefully delete loose files from previous runs

(to address test failing after forces stop)

Changeset: f2595c32cad71d5ecd35693352d35a9e30b6a068

[Alex Buts]

refs #6362 deleting files for loader

Changeset: de553d27fdde386d129e8582111fa02b87360215

[Alex Buts]

refs #6362 disabled all failing tests except one

trying to see what is going on

Changeset: b88385adb2b47c1566906a38e2289f19b38ca55a

[Alex Buts]

refs #6362 Not calling Nexus writer when no events to write

Changeset: 348193860ae77420e41e577956692ecaeffb39d7

[Alex Buts]

refs #6362 Redefined nexus writer and enabled couple of tests

, failing earlier

Changeset: c6c5ce26d7eb465c7b7538092bd88034cc53db6c

[Alex Buts]

refs #6362 Syntax error

Changeset: d63d42f2c0dba47d9fab3d3474306ff0a3270218

[Alex Buts]

refs #6362 Trying to fix/disable the failing tests

Changeset: bdc863ceecb305b6eac520c91bdd329a6a523024

[Alex Buts]

refs #6362 Enabled one previously failing test to see what happens

Changeset: c126b1df5b83e97b00f6fa8a894324acc86a278f

[Alex Buts]

refs #6362 Enabled another failing test

Changeset: 68d8fc39f1b6531b81cadaeb3ca6dffbb6aa3af0

[Alex Buts]

refs #6362 Enabling remaining failing tests

Changeset: ef907c2ac69890b6f5d1e0b2064feecb7d3ed56f

[Alex Buts]

refs #6362 unblocking some performance tests (not all yet)

Changeset: 2f55d5f8581ddd5dd14c6a5630d42c081e2758c4

[Alex Buts]

The main purpose of this ticket is achieved, though the structure of the MDBoxes is not yet optimal. But main things that done are:

1) ISaveable/DiskBuffer are mainly responsible for bookkeeping for file-based operations delegating particular object specific operations to children. 2) Controlling parameters are reduced to 4 and user should not set them alone. I hope their meaning is more transparent as usually they are set up by appropriate IO operations.

It is easy/difficult to test this ticket. If all tests and MD algorithms are working fine, which should be obvious to the time of testing, the changes are correct.

A code review and suggestions on further improvements would be useful.

[Martyn Gigg]

Ticket #5849 related to file-based MD workspaces no functioning correctly and it would appear that this works correctly now. The unit & system tests are fine also.

There has been an issue regarding performance. Ticket #6513

[Stuart Campbell]

This ticket has been transferred to github issue ​7208