[nSLUG] nSLUG Digest, Vol 575, Issue 1

chris thompson ct8ball at gmail.com
Wed Jun 3 15:07:02 ADT 2015


Thank you Hatem. I'm moving forward dbless but your feedback certainly is
appreciated.

On Tue, Jun 2, 2015, 11:55 AM  <nslug-request at nslug.ns.ca> wrote:

> Send nSLUG mailing list submissions to
>         nslug at nslug.ns.ca
>
> To subscribe or unsubscribe via the World Wide Web, visit
>         http://nslug.ns.ca/mailman/listinfo/nslug
> or, via email, send a message with subject or body 'help' to
>         nslug-request at nslug.ns.ca
>
> You can reach the person managing the list at
>         nslug-owner at nslug.ns.ca
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of nSLUG digest..."
>
>
> Today's Topics:
>
>    1.  BLOB's and Large File Storage (chris thompson)
>    2. Re:  BLOB's and Large File Storage (Hatem Nassrat)
>    3. Re:  BLOB's and Large File Storage (Stephen Gregory)
>    4. Re:  nSLUG Digest, Vol 574, Issue 1 (chris thompson)
>    5. Re:  BLOB's and Large File Storage (George N. White III)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Mon, 01 Jun 2015 17:51:41 +0000
> From: chris thompson <ct8ball at gmail.com>
> To: "nslug at nslug.ns.ca" <nslug at nslug.ns.ca>
> Subject: [nSLUG] BLOB's and Large File Storage
> Message-ID:
>         <
> CAE1Cd4FE0_5Asc+JLdqNk+RQH24pK4hSWnek2qW6AZO+dQvjhw at mail.gmail.com>
> Content-Type: text/plain; charset="utf-8"
>
> Happy Monday,
>
> Large File storage(Pictures). Is it of any benefit to store such files in a
> DB via BLOB? or alternately what in your opinion is the best way to store
> and share these on a local network? file-system play any part here? I'm
> very interested to hear the feedback from you folks as I'm trying to
> architect a mechanism for large file versioning if you will.... aka svn/cvs
> for large files, I know there are a few that exist, but I believe a
> ground-up approach needs to be taken to maximize the effectiveness of any
> solution.
>
> Try not to get duck feet out there.
>
> Chris
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: <
> http://nslug.ns.ca/pipermail/nslug/attachments/20150601/74405fb8/attachment-0001.html
> >
>
> ------------------------------
>
> Message: 2
> Date: Mon, 01 Jun 2015 18:35:36 +0000
> From: Hatem Nassrat <hnassrat at gmail.com>
> To: Nova Scotia Linux User Group <nslug at nslug.ns.ca>
> Subject: Re: [nSLUG] BLOB's and Large File Storage
> Message-ID:
>         <
> CACFAeE0my9DH0QAoEdA-O4EZ7TSDFjj4NEhM_VeHhi-+3tOdtA at mail.gmail.com>
> Content-Type: text/plain; charset="utf-8"
>
> I have a question/clarification regarding your last statement.
>
> On Mon, Jun 1, 2015 at 2:51 PM chris thompson <ct8ball at gmail.com> wrote:
>
> > I know there are a few that exist, but I believe a ground-up approach
> > needs to be taken to maximize the effectiveness of any solution.
> >
>
> I believe you meant, none of the solutions you have seen had shown any
> promise into solving the problem you were tackling. One should always try
> to build on other software / systems rather than start from scratch. For
> example, no one today would try to come up with their own operating system
> to make sure it is extra efficient for their users' use case. Usually they
> would customize the operating system / kernels as needed / if needed / when
> needed).
>
>
> > Large File storage(Pictures). Is it of any benefit to store such files in
> > a DB via BLOB? or alternately what in your opinion is the best way to
> store
> > and share these on a local network? file-system play any part here? I'm
> very
> >
>
> So back to the first point, this really depends on what is it you are
> trying to do. This question seems to ask what is the best way to store
> images for your app to access. A filesystem is a database so really its all
> the same. It depends on your preference on how to access and use these
> files in your application. A natural extension to this question is which
> filesystem or which database should I use? There might be DBs out there
> tuned to store images, but in general they should be all the same. If I was
> building this I would probably go the DB route since I can attach much more
> metadata to a "file" than merely a name (of course you can also hack that
> up on the filesystem if you wanted to).
>
> interested to hear the feedback from you folks as I'm trying to architect a
> > mechanism for large file versioning if you will.... aka svn/cvs for large
> > files, I know there are a few that exist,
> >
>
> I would look at what already exists for image versioning and how they go
> about updating and merging versions. Just storing the files as simple blobs
> might not allow you to do things like detecting what's changed (you might
> need a 2d array sort of structure to store your data for example). You
> really need to think about the actions you are going to allow on your data
> to find the best data structure to lay out your data and hence the storage
> mechanism.
>
> I would look at object storage databases, such as HDF5 or even Document
> Store DBs such as MongoDB.
>
> -- Hatem
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: <
> http://nslug.ns.ca/pipermail/nslug/attachments/20150601/1c023999/attachment-0001.html
> >
>
> ------------------------------
>
> Message: 3
> Date: Mon, 1 Jun 2015 14:40:58 -0400
> From: Stephen Gregory <nslug at kernelpanic.ca>
> To: Nova Scotia Linux User Group <nslug at nslug.ns.ca>
> Subject: Re: [nSLUG] BLOB's and Large File Storage
> Message-ID:
>         <
> CAJ5K_5pX0xQYV1mSG4QL5eSNDBbMbHNEXs5Ynspwxr4oUCDShw at mail.gmail.com>
> Content-Type: text/plain; charset=UTF-8
>
> My friendly neighbourhood DBA says "Don't use blobs for file storage."
> Store a pointer to the file (accessible via nfs, web, samba, etc).
>
> Also: don't reinvent the wheel. I believe there is a git extension to
> manage and share binary files.
>
> --
> sg
>
>
> On Mon, Jun 1, 2015 at 1:51 PM, chris thompson <ct8ball at gmail.com> wrote:
> > Happy Monday,
> >
> > Large File storage(Pictures). Is it of any benefit to store such files
> in a
> > DB via BLOB? or alternately what in your opinion is the best way to store
> > and share these on a local network? file-system play any part here? I'm
> very
> > interested to hear the feedback from you folks as I'm trying to
> architect a
> > mechanism for large file versioning if you will.... aka svn/cvs for large
> > files, I know there are a few that exist, but I believe a ground-up
> approach
> > needs to be taken to maximize the effectiveness of any solution.
> >
> > Try not to get duck feet out there.
> >
> > Chris
> >
> > _______________________________________________
> > nSLUG mailing list
> > nSLUG at nslug.ns.ca
> > http://nslug.ns.ca/mailman/listinfo/nslug
> >
>
>
> ------------------------------
>
> Message: 4
> Date: Mon, 01 Jun 2015 19:45:54 +0000
> From: chris thompson <ct8ball at gmail.com>
> To: nslug at nslug.ns.ca
> Subject: Re: [nSLUG] nSLUG Digest, Vol 574, Issue 1
> Message-ID:
>         <
> CAE1Cd4HqTUy9SRHYTn5nPVqvbey71q07jcUhPYervWjKo2Xvrw at mail.gmail.com>
> Content-Type: text/plain; charset="utf-8"
>
> Hello All,
>
> Back in the day I was in a few times for the meetings, and must apologize,
> I have been out of the game for a bit and out of work. But now I'm back
> into it, and so is my wife, so it's quite the opposite of too much time not
> enough money. I do however intend on making it to the next meeting...
>
> I'm going to stick with, "If there was an announcement earlier I would have
> come", that way I can continue to blame Ben.
>
> Sincerely,
> Chris
>
> On Mon, Jun 1, 2015 at 11:55 AM <nslug-request at nslug.ns.ca> wrote:
>
> > Send nSLUG mailing list submissions to
> >         nslug at nslug.ns.ca
> >
> > To subscribe or unsubscribe via the World Wide Web, visit
> >         http://nslug.ns.ca/mailman/listinfo/nslug
> > or, via email, send a message with subject or body 'help' to
> >         nslug-request at nslug.ns.ca
> >
> > You can reach the person managing the list at
> >         nslug-owner at nslug.ns.ca
> >
> > When replying, please edit your Subject line so it is more specific
> > than "Re: Contents of nSLUG digest..."
> >
> >
> > Today's Topics:
> >
> >    1.  June Meeting (David Rodgers)
> >    2. Re:  June Meeting (Ben Armstrong)
> >
> >
> > ----------------------------------------------------------------------
> >
> > Message: 1
> > Date: Sun, 31 May 2015 14:14:14 -0300
> > From: David Rodgers <davidfrodgers at gmail.com>
> > To: nslug at nslug.ns.ca
> > Subject: [nSLUG] June Meeting
> > Message-ID: <556B4166.1090700 at gmail.com>
> > Content-Type: text/plain; charset=utf-8
> >
> > Will there be a meeting tomorrow, Monday June 1?
> >
> > Thanks,
> > David
> >
> >
> > ------------------------------
> >
> > Message: 2
> > Date: Sun, 31 May 2015 18:56:40 -0300
> > From: Ben Armstrong <synrg at sanctuary.nslug.ns.ca>
> > To: Nova Scotia Linux User Group <nslug at nslug.ns.ca>
> > Subject: Re: [nSLUG] June Meeting
> > Message-ID: <556B8398.5040903 at sanctuary.nslug.ns.ca>
> > Content-Type: text/plain; charset=windows-1252
> >
> > On 31/05/15 02:14 PM, David Rodgers wrote:
> > > Will there be a meeting tomorrow, Monday June 1?
> > >
> >
> > Yes. Sorry I've gotten lax about announcements. I've not heard anything
> > about a tech talk, so in its place there will be a meeting tomorrow,
> > June 1, at Uncommon Grounds on South Park St.
> >
> > Ben
> >
> >
> >
> > ------------------------------
> >
> > Subject: Digest Footer
> >
> > _______________________________________________
> > nSLUG mailing list
> > nSLUG at nslug.ns.ca
> > http://nslug.ns.ca/mailman/listinfo/nslug
> >
> >
> > ------------------------------
> >
> > End of nSLUG Digest, Vol 574, Issue 1
> > *************************************
> >
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: <
> http://nslug.ns.ca/pipermail/nslug/attachments/20150601/0af4ef6c/attachment-0001.html
> >
>
> ------------------------------
>
> Message: 5
> Date: Mon, 1 Jun 2015 20:03:51 -0300
> From: "George N. White III" <gnwiii at gmail.com>
> To: Nova Scotia Linux User Group <nslug at nslug.ns.ca>
> Subject: Re: [nSLUG] BLOB's and Large File Storage
> Message-ID:
>         <CAKTOP44pt5p_mKJ4+qcG1W-95AVJkpzxUkEHL4_f-1pt_ked=
> A at mail.gmail.com>
> Content-Type: text/plain; charset="utf-8"
>
> On Mon, Jun 1, 2015 at 2:51 PM, chris thompson <ct8ball at gmail.com> wrote:
>
> >
> > Happy Monday,
> >
> > Large File storage(Pictures). Is it of any benefit to store such files in
> > a DB via BLOB? or alternately what in your opinion is the best way to
> store
> > and share these on a local network? file-system play any part here? I'm
> > very interested to hear the feedback from you folks as I'm trying to
> > architect a mechanism for large file versioning if you will.... aka
> svn/cvs
> > for large files, I know there are a few that exist, but I believe a
> > ground-up approach needs to be taken to maximize the effectiveness of any
> > solution.
> >
> > Try not to get duck feet out there.
> >
>
> Depends on how the files will be used.  I should mention hdf5 or netcdf4
> formats with OpenDAP clientsand servers.  This supports discovery and
> processing applications (Matlab and many others).
>
> At work we are redesigning a legacy workflow that used externally
> compressed hdf4 files with one based on netcdf4 with internal compression.
> With the old workflow, users had to download the compressed files and
> decompress for use, even if they wanted only a small subset of a file.  The
> new workflow should allow users to load a variety of subsets directly into
> applications.  The catch seems to be managing the metadata, but it is hard
> to imagine that it could be worse than the old system where users often
> needed lengthy email exchanges to get the metadata in a form they could
> use.
>
> --
> George N. White III <aa056 at chebucto.ns.ca>
> Head of St. Margarets Bay, Nova Scotia
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: <
> http://nslug.ns.ca/pipermail/nslug/attachments/20150601/bbd1bc21/attachment-0001.html
> >
>
> ------------------------------
>
> Subject: Digest Footer
>
> _______________________________________________
> nSLUG mailing list
> nSLUG at nslug.ns.ca
> http://nslug.ns.ca/mailman/listinfo/nslug
>
>
> ------------------------------
>
> End of nSLUG Digest, Vol 575, Issue 1
> *************************************
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://nslug.ns.ca/pipermail/nslug/attachments/20150603/3b43c49d/attachment-0001.html>


More information about the nSLUG mailing list