Guys this is really cool! I'm following the code only on the surface but I understand that file-backed caching will soon be supported? If this is interesting I can also take a look at HDF5 based caching in the future, because the file format supports basically everything that an N-dimensional data structure could need.

On the other hand, xtensor already supports HDF5 as a backend so maybe its overkill to add dedicated support here (again)?

file-backed caching will soon be supported?

xdisk_array indeed acts as a file-backed cached array.

On the other hand, xtensor already supports HDF5 as a backend so maybe its overkill to add dedicated support here (again)?

This PR gets us closer to Zarr, I'm not sure about HDF5.

There are two orthogonal things here:

• "chunk array" and "file array": these features are not bound to any file format or spec implementation, they are generic enough to live in xtensor core.
• the different file formats (HDF5, zarr), which should be implemented either in xtensor-io, or in a dedicated package.

EDIT: replace "disk array" with "file array" according to the comment below.

@JohanMabille I think xfiles_array is the one which is generic: it should manage a pool of file-backed arrays (currently the pool only consists of one file array). It doesn't know what these file arrays are, they can live on disk or in the cloud.
xdisk_array is an on-disk file array. In the current PR, I used a CSV file format just to make debugging easy.The xdisk_array is passed a file and an in-memory container for its content (an xarray) from xfiles_array.
I'm not sure this is the right architecture though, any feedback is very welcome.

Alright, so maybe the disk array will be replaced by an in-memory container + a serializer passed directly to the file array class. Let's see what's required in the disk array while you complete itsimplementation, and if the class is really required or not.

The new test fails with the following error:

unknown file: Failure
C++ exception with description "stod" thrown in the test body.

which happens as soon as a file has to be written. It works fine outside the test environment. Thoughts?

Alright, so maybe the disk array will be replaced by an in-memory container + a serializer passed directly to the file array class.

The issue I see with that is that xfiles_array doesn't know if the array is being assigned, the assignment takes place in xdisk_reference's operator=(). At the same time, this is not great either because there is no "chunk view", only an "element view", which means that as soon as a single element is assigned the whole chunk is saved.

Another possibility is to flag the xdisk_array as "dirty" when an element is assigned, and save to disk only upon destruction / unload.

Not sure about the failing test:

/home/travis/build/xtensor-stack/xtensor/test/test_xmanipulation.cpp:121: Failure
Expected equality of these values:
  fl
    Which is: { 1, 2, 1, 11, 12, 13, 21, 22, 23, 51, 52, 53, 61, 62, 63, 71, 72, 73 }
  expected
    Which is: { 1, 2, 3, 11, 12, 13, 21, 22, 23, 51, 52, 53, 61, 62, 63, 71, 72, 73 }

Just thinking, currently a xdisk_array doesn't work on its own because it expects an xarray and a file created in xfiles_array, which manages a pool of arrays. But it might be interesting to have an xdisk_array that embeds its own xarray and file, so that it can be used as a persistent array. It would have to implement all the semantic stuff though.

Now that xdisk_array has nothing to do with an on-disk array, it should be renamed. It is basically a wrapper around an array notifying when the array is changed. Any idea?
xfiles_array does all the file loading/dumping. It really makes sense only in the context of a chunked array. We should also find a better name.

What about the following:

• xfile_array: a N-D array backed by a file on the disk. It is a full expression (so it can be assigned other expressions) and can be used outside of the context of chunk arrayst. It embeds an in-memory container (the cache), a flag to say if the cache is dirty, and another flag to say if the file is already loaded in memory.. It also provides explicit load / unload methods to open the file and put its content in the in-memory container / write the in-memory container to the file and free its memory. Ideally, it is templated by a serializer / deserializer so it is independent form the file format.

• xchunk_loading_manager: a container holding xfile_array objects (or any expression providing explicit load / unload methods) managing the load / unload of chunks depending on their size and the maximal amount of memory that a chunk array can use, and which chunk is accessed.

Notice that xfile_array assumes that the file can be entirely loaded into memory, we may need another mechanism for large files (maybe a new xmaped_file_array class that will rely on memory mapping).

Thanks for the suggestions. I think it is a better abstraction than what we have right now, where we have the file loading/dumping in xfiles_array (that you now call xchunk_loading_manager), but should really be in xdisk_array (that you now call xfile_array).
A couple of remarks:

• maybe we should introduce another abstraction in xfile_array, which doesn't have to be an on-disk array, but could be e.g. a cloud-based array. It could be through a template file type, providing that this type accepts common methods like open and close.
• maybe change xchunk_loading_manager to xchunk_store_manager?

For the first point, that could be handled by a io_type template parameter (or io_handler), which implements open / close methods. I agree that this will avoid to reinvent the wheel when we want to support distant files. Therefore, the declaration of xfile_array would look like:

template <class T, class io_handler, class serializer>
class xfile_array;

Totally agree on the second point, xchunk_store_manager is a better name.

@JohanMabille This last commit renames xfiles_array to xchunk_store_manager, and xdisk_array to xfile_array. It also moves the file-related logic to the new xfile_array.
For now the file abstraction is handled by the SIN and SOUT template parameters (e.g. ifstream and ofstream for disk files):

template <class EC, class SIN, class SOUT>
class xfile_array;

I will look at your suggestion later.
Also, xfile_array is not a full expression yet. And xchunk_store_manager only manages one xfile_object for now, I need to implement a pool of objects.

@davidbrochart awesome! I think further improvements (such as making xfile_array a valid tensor expression, or making xchunk_store_manager handling a pool) can be done in dedicated PRs.

I would also gather SIN and SOUT in a single io_handler tempate paremeter that handles both reading and writing.