[nSLUG] BLOB's and Large File Storage

Hatem Nassrat hnassrat at gmail.com
Mon Jun 1 15:35:36 ADT 2015


I have a question/clarification regarding your last statement.

On Mon, Jun 1, 2015 at 2:51 PM chris thompson <ct8ball at gmail.com> wrote:

> I know there are a few that exist, but I believe a ground-up approach
> needs to be taken to maximize the effectiveness of any solution.
>

I believe you meant, none of the solutions you have seen had shown any
promise into solving the problem you were tackling. One should always try
to build on other software / systems rather than start from scratch. For
example, no one today would try to come up with their own operating system
to make sure it is extra efficient for their users' use case. Usually they
would customize the operating system / kernels as needed / if needed / when
needed).


> Large File storage(Pictures). Is it of any benefit to store such files in
> a DB via BLOB? or alternately what in your opinion is the best way to store
> and share these on a local network? file-system play any part here? I'm very
>

So back to the first point, this really depends on what is it you are
trying to do. This question seems to ask what is the best way to store
images for your app to access. A filesystem is a database so really its all
the same. It depends on your preference on how to access and use these
files in your application. A natural extension to this question is which
filesystem or which database should I use? There might be DBs out there
tuned to store images, but in general they should be all the same. If I was
building this I would probably go the DB route since I can attach much more
metadata to a "file" than merely a name (of course you can also hack that
up on the filesystem if you wanted to).

interested to hear the feedback from you folks as I'm trying to architect a
> mechanism for large file versioning if you will.... aka svn/cvs for large
> files, I know there are a few that exist,
>

I would look at what already exists for image versioning and how they go
about updating and merging versions. Just storing the files as simple blobs
might not allow you to do things like detecting what's changed (you might
need a 2d array sort of structure to store your data for example). You
really need to think about the actions you are going to allow on your data
to find the best data structure to lay out your data and hence the storage
mechanism.

I would look at object storage databases, such as HDF5 or even Document
Store DBs such as MongoDB.

-- Hatem
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://nslug.ns.ca/pipermail/nslug/attachments/20150601/1c023999/attachment.html>


More information about the nSLUG mailing list