FileOutputStream questions

[Roedy Green]

I have just realised I do not understand some basic things about
unbuffered FileOutputStream.

I used them in two ways, wrapping in a BufferedOutputStream or
BufferedWriter, or writing an entire file in one i/o without
buffering.

If you write single bytes at a time, will you trigger physical I/O on
every byte, or is there some small buffer in there anyway?

I have experimented with flush during file write. The file size stays
at 0 until I close, at least to DIR.

What I want is to log bytes that will be largely recoverable even if
the program terminates unexpectedly without closing. Flush does not do
it. Close/reopen periodically seem at bit heavy handed. Is there
something I am missing?

[markspace]

Post by Roedy Green
I have experimented with flush during file write. The file size stays
at 0 until I close, at least to DIR.

What's DIR? The directory?

[Roedy Green]

On Tue, 25 Dec 2012 07:54:07 -0800, markspace

Post by markspace
What's DIR? The directory?

the Windows command.exe/TCMD.exe intrinsic DIR directory command.

If I type dir somefile.bin during the generation of somefile.bin with
a FileOutputStream I see size 0 until close.

[Jukka Lahtinen]

Post by Roedy Green
I have just realised I do not understand some basic things about
unbuffered FileOutputStream.
I used them in two ways, wrapping in a BufferedOutputStream or
BufferedWriter, or writing an entire file in one i/o without
buffering.
If you write single bytes at a time, will you trigger physical I/O on
every byte, or is there some small buffer in there anyway?

I suppose that even if you use a non-buffering java class for file
writing, the operating system may still buffer the output. AFAIK, at
least Linux does that.

Post by Roedy Green
I have experimented with flush during file write. The file size stays
at 0 until I close, at least to DIR.

The directory entry is probably not fully written or updated before you
close the fle. As long as the file is open for writing, the operating
system does not know wheter you will still be writing to it.

[Knute Johnson]

Post by Roedy Green
I have just realised I do not understand some basic things about
unbuffered FileOutputStream.
I used them in two ways, wrapping in a BufferedOutputStream or
BufferedWriter, or writing an entire file in one i/o without
buffering.
If you write single bytes at a time, will you trigger physical I/O on
every byte, or is there some small buffer in there anyway?
I have experimented with flush during file write. The file size stays
at 0 until I close, at least to DIR.
What I want is to log bytes that will be largely recoverable even if
the program terminates unexpectedly without closing. Flush does not do
it. Close/reopen periodically seem at bit heavy handed. Is there
something I am missing?

I use a buffered output stream with the file opened in append mode to
keep some log files. I flush those after every write.

"If the intended destination of this stream is an abstraction provided
by the underlying operating system, for example a file, then flushing
the stream guarantees only that bytes previously written to the stream
are passed to the operating system for writing; it does not guarantee
that they are actually written to a physical device such as a disk drive."

I haven't had any problems with the above getting it to write
immediately. All of this has been on WinXP which I assume you probably
are with the mention of DIR.

Under Windows, even the OS has a write buffer. I don't know what causes
it to commit or if a Java stream flush is passed through. I just looked
at the source code and OutputStream.flush() doesn't do anything.

[Roedy Green]

On Tue, 25 Dec 2012 11:45:30 -0800, Knute Johnson

Post by Knute Johnson
I use a buffered output stream with the file opened in append mode to
keep some log files. I flush those after every write.

Have you ever aborted or powered off to see how much you lose?

I suspect flush is not really much help.

See http://mindprod.com/jgloss/flush.html for my findings.

[Knute Johnson]

Post by Roedy Green
On Tue, 25 Dec 2012 11:45:30 -0800, Knute Johnson

Post by Knute Johnson
I use a buffered output stream with the file opened in append mode to
keep some log files. I flush those after every write.

Have you ever aborted or powered off to see how much you lose?

No but I have copied the file while it was open and it didn't appear to
lose any data.

Post by Roedy Green
I suspect flush is not really much help.

The buffers are flushed anyway.

[Knute Johnson]

Post by Roedy Green
On Tue, 25 Dec 2012 11:45:30 -0800, Knute Johnson

Post by Knute Johnson
I use a buffered output stream with the file opened in append mode to
keep some log files. I flush those after every write.

Have you ever aborted or powered off to see how much you lose?

No but I have copied the file while it was open and it didn't appear to
lose any data.

Post by Roedy Green
I suspect flush is not really much help.

The buffers are flushed anyway.

[Roedy Green]

On Tue, 25 Dec 2012 06:36:29 -0800, Roedy Green

Post by Roedy Green
What I want is to log bytes that will be largely recoverable even if
the program terminates unexpectedly without closing. Flush does not do
it. Close/reopen periodically seem at bit heavy handed. Is there
something I am missing?

I discovered this

fos.flush();
fos.getFD().sync();

Which acts like close/reopen.

[Knute Johnson]

Post by Roedy Green
On Tue, 25 Dec 2012 06:36:29 -0800, Roedy Green

Post by Roedy Green
What I want is to log bytes that will be largely recoverable even if
the program terminates unexpectedly without closing. Flush does not do
it. Close/reopen periodically seem at bit heavy handed. Is there
something I am missing?

I discovered this
fos.flush();
fos.getFD().sync();
Which acts like close/reopen.

I never saw that before and it has been there since almost the
beginning. Post back if that works for you please, cause I'm really
curious now.

[Roedy Green]

On Tue, 25 Dec 2012 14:05:52 -0800, Knute Johnson

Post by Knute Johnson
I never saw that before and it has been there since almost the
beginning. Post back if that works for you please, cause I'm really
curious now.

It does work, exactly as you would expect. When I flush sync, I can
see a burst of disk activity, and when I look with dir, the file has
grown. There is not much point in getting bytes on disk if the length
in the directory is not updated too unless you were prepared to do
some low level recovery after a crash, or unless you allocated your
space, then closed, the opened, then just used flush, keeping track of
length by your own internal means.

I remember doing this sort of thing for database recovery back in the
mainframe days.

[markspace]

Post by Knute Johnson
I never saw that before and it has been there since almost the
beginning. Post back if that works for you please, cause I'm really
curious now.

Googled for "java flush doesn't work", third result:

<http://stackoverflow.com/questions/11829922/logback-file-appender-doesnt-flush-immediately>

No, I didn't know about it either.

[markspace]

Post by Knute Johnson
I never saw that before and it has been there since almost the
beginning. Post back if that works for you please, cause I'm really
curious now.

Googled for "java flush doesn't work", third result:

<http://stackoverflow.com/questions/11829922/logback-file-appender-doesnt-flush-immediately>

No, I didn't know about it either.

[Stanimir Stamenkov]

Post by Roedy Green
I discovered this
fos.flush();
fos.getFD().sync();
Which acts like close/reopen.

You may also checkout the following thread from the Xerces-J Users
mailing list:

"Does LSSerailzer creates a thread of its own."
<http://mail-archives.apache.org/mod_mbox/xerces-j-users/200708.mbox/browser>

The following reply suggest a newer Java NIO API -
FileChannel.force(true), for the same purpose:

http://mail-archives.apache.org/mod_mbox/xerces-j-users/200708.mbox/%3C46D3FBFA.8060104%40netscape.net%3E

http://docs.oracle.com/javase/6/docs/api/java/nio/channels/FileChannel.html#force%28boolean%29

And the following appears just the same as your finding:

http://mail-archives.apache.org/mod_mbox/xerces-j-users/200708.mbox/%3C46D59744.2010505%40netscape.net%3E