MDCS - how is data handled?

Within an MDCS system, is the data sent across the network and only sits in the memory of the head/workers, or is there a requirement for data storage?

Answer 1

[Harsha Medikonda]

Based on my understanding you wish to know the data storage requirements for clusters. Please refer to the following link for the requirements of data storage Requirements for data storage on cluster computers

To transfer data between workers, you can use 'addAttachedFiles'. When files are attached using this command, they are shared across all the workers. Currently there is a 2 GB limit for the data transfer across workers. Refer to the following link for more information on 'addAttachedFiles'

addattachedfiles

Answer 2

[Lachlan Simpson]

Thanks for your reply Harsa.

I was wondering if I could get some clarification. When the documentation says:

    A shared file system between user desktops and cluster is strongly 
    recommended. Availability of a shared file system is assumed by default 
    for all the built-in configurations. An API is available to extend these 
    configurations for environments with nonshared file systems.

I am unclear what this means.

Is the suggestion that:

• the entire filesystem should be shared (!)
• /home
• some sort of scratch space - /mnt/DATA or somesuch
• /usr/local/MATLAB/*
• /usr/local/MATLAB/R20xyz/toolbox/*

Some combination of the above?