File Storage:

Introduction: To help with data file management some practices are being implemented.  These practices will make data files more easily available to VS200/VS120 users at a server location which is accessible on campus the network. 

Overview: In order to backup users' data files on the D: drive of the Olympus VS200 scanner will be copied to the data server \\minsky.med.unc.edu\vs200 nightly at ~4 AM.  This will make files on the D: drive of the VS200 system available on the \\minsky.med.unc.edu\vs200 server the next day.  The files on the server are accessible over the network using the usual shared directory mapping procedures and an mhmicroscopy domain account. Thus it will not be necessary to be at the VS200 system to access data on the next day after scanning.

What is being done?

• Every evening files will be automatically copied from the D: drive if they do not exist at \\minsky.med.unc.edu\vs200 or if they are newer than the ones already at \\minsky.med.unc.edu\vs200.  Date & time stamps for files and directories will be updated at the server as well.  Files will remain on the D: drive until removed by the owner or after 3 months, or more if space is available, by the system manager.

What are the data implications for users?

• Files will be at 2 locations initially
• The copy process is cumulative.  Files which are newer on the D: drive & files on the D: drive that do not exist at \\minsky.med.unc.edu\vs200 server will be copied nightly
• Deletion of files at the VS200 D: drive will leave copies on the \\minsky.med.unc.edu\vs200 server and also on a hidden backup server

What are the limitations?

• The VS200 C: & E: drive files are not copied to the server
• Newer files will displace older files with the same name in each respective directory on the \\minsky.med.unc.edu\vs200  server storage space
• No archiving is carried out. Older files with the same name as newer files in each respective directory will be overwritten and will not be recoverable
• At some future time someone will have make the executive decision to remove very old data on the server which will inevitably fill up
• Files deleted from the D: drive will remain at \\minsky.med.unc.edu\vs200.  This is good since the data is effectively backed up. 
• Files deleted from the D: drive will remain at \\minsky.med.unc.edu\vs200.  This is also bad since files deleted from D: will remain on the server and may be unwanted, and, hence, take up valuable server storage space.
• If files are deleted from  \\minsky.med.unc.edu\vs200  they should be deleted from the VS200 D: drive as well.  This represents an extra step for users.  If the files are not also deleted from D: they will reappear at \\minsky.med.unc.edu\vs200 after the next evenings copy process.
• Copying will slow down slide scans being run ~4 AM.  The slow down should only have a minor impact on performance unless scans are being saved to a network drive.
• This is all necessarily rather complicated.  But it solves the D: drive filling up problem and preserves files which may be needed by users who have not copied their data off the system D: drive.

What will be done with data on the D: drive?

• Files older than ~1 month will be subject to summary deletion on the D: drive after it is made sure copies exist on the \\minsky.med.unc.edu server or potentially another backup server.

What users can do to help with data management?

• If files are deleted from the \\minsky.med.unc.edu\vs200 they should also be deleted from the VS200 D: drive.  This represents an extra step for users.  A work around is to add "-DELETE" to the top directory or directories on the D: if the data is not needed in both the D: drive and the \\minsky.med.unc.edu\vs200 server (It will be up to management to notice the message and handle deletion of all files along the storage chain.  This minor overhead has the benefit of letting data managers know the files are not needed and hence can be deleted along the server storage chain)
• Do not rename subdirectory names on the D: drive since that will create two directories on the \\minsky.med.unc.edu server creating duplicated data and wasting storage space.  Exception is adding "-DELETE".

General Data Storage Notes