[CLUE-Tech] Re: [CO-SAGE] how to handle open files - upload server

[Steve Sayler]

how about sending a "upload-completed"  file just after sending the data file.
The processing cron job would first look for the presence of the
"upload-completed"  file before beginning the processing.  Remove the
"upload-completed"  file during the processing procedure.



"Alek O. Komarnitsky (N-CSC)" wrote:

> > From co-sage-admin at fpcc.net Tue Jan 13 09:51 MST 2004
> > From: Nate Duehr <nate at natetech.com>
> >
> > [Sorry about the cross-post for those on all the lists, just attempting to
> > cover a wider audience.]
> >
> > Here's an interesting one, or maybe I'm just not creative enough today.  I
> > have a Linux server where people upload some files to me for processing
> > (using sftp).  The files show up, I move them to another machine to do
> > the dirty work.  Simple, right?  Move those files elsewhere after they
> > arrive, which can pretty easily be done from a cron job, no problem
> > there.
> >
> > The case I'm wondering about: How to detect in that regular cron job if
> > the file is open (and probably growing) right at the time because it's
> > being uploaded right then... so the cron doesn't attempt to move it to
> > the other machine in the middle of an upload.
> >
> > I'm thinking some silly hack at the top of the "move to other machine"
> > script that would look at the directory contents with lsof to make sure
> > nothing has any of the files to be moved already open.  Is there a
> > simpler/smarter way?
>
> Here's some quick ideas for hacks:
>    1.  If it's not time critical, look at the modify time.
>        If withen the last minute, skip it ... and then have it
>        picked up in the cron run.
>
>    2.  Do a stat to get file size, then sleep for 5 seconds, and do a
>        stat again ... if file size has changed, skip it.
>
>
> alek
>
> P.S. FYI FWIW: I got some Emails from folks on this list who enjoyed
>      my Christmas Lights/Webcam - the final wrapup has been done - see:
>         http://www.komar.org/xmas/
>      Click on the 2003 link and then stats for those interested in
>      that type of stuff - also includes an analysis of the Slashdot Effect
>      on Christmas Lights. There's a 2004/what's-new link from the top-level;
>      this is a pretty technical/astute crowd, so any ideas appreciated.
> -
> This is the CO-SAGE Mailing list.  Please don't post attachments,
> binaries or excessively long posts.  To unsubscribe, please visit
> http://lists.fpcc.net/mailman/listinfo/co-sage

--
         _______________________________________________________
        |                  Steve Sayler                         |
        |US Gov't, Department of Commerce,   Tel: 303-497-3959  |
        |NOAA, Space Environment Center      Fax: 303-497-3645  |
      _ |325 Broadway, MS: R/E/SE2                              | _
     / )|Boulder, Colorado, USA 80303                           |( \
    / / |                                                       | \ \
  _( (_ |  _         email: steve.sayler at noaa.gov            _  | _) )_
 (((\ \>|_/ >_______________________________________________< \_|</ /)))
 (\\\\ \_/ /                               __                \ \_/ ////)
  \       /     _ __   ___   __ _  __ _   / /__  ___  ___     \       /
   \    _/     | '_ \ / _ \ / _` |/ _` | / / __|/ _ \/ __|     \_    /
   /   /       | | | | (_) | (_| | (_| |/ /\__ \  __/ (__        \   \
               |_| |_|\___/ \__,_|\__,_/_/ |___/\___|\___|


-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://cluedenver.org/pipermail/clue-tech/attachments/20040113/08222d53/attachment.html