Having problems with message search?
Thanks Jim.
I understand that there are several ways round this.
However...
This problem came upon me unawares, and could do the same to anyone at any time.
A utility of the calibre of PFGW should not be so loosely coded that it can write unkillable files of more than 1 Gb each, which I have had to quarantine because any attempt to delete them locks the operating system.
I have so far permanently lost 2.5 Gb of my hard disk capacility to this bug.
chkdsk is unable to recover the lost file space.
Can this rogue behaviour not be relatively easily checked for, and a graceful error message be given?
Mike
PS I stated file sizes a bit incorrectly: the input file was about 2Gb, not 4Gb; the limiting pfgw.log size was 1.5 Gb on one occasion, and 1.1 Gb on another. So it looks as if 4-Gb limits were not being closely approached, let alone exceeded. I repeat: the input file was read completely and ok; the problem lies in the output file i/o.
-----Original Message-----
From: jim_fougeron <jfoug@...>
To: openpfgw@yahoogroups.com
Sent: Tue, 6 Jun 2006 16:09:15 -0000
Subject: [openpfgw] Re: Bug in processing huge input files from NewPGen
OpenPFGW (win32) uses FILE structures, which under VC (and MinGW),
uses a 32 bit max sized file. I would assume that 70m values in the
pfgw.log file pushed over a 4GB level.
Your only alternative, is to cut the file up.
Jim.
--- In openpfgw@yahoogroups.com , mikeoakes2@... wrote:
>
> I believe there is a problem with writing exceptionally long
pfgw.log files.
>
> I am using the command line
> pfgw -b107 xxx.inp
> with an input file coming from NewPGen, and starting:-
> 33319550988:P:0:2:65
> 2 71
> 3 71
> ...
> This file is 4 Gb in size, and has 180,259,222 lines.
>
> The pfgw.log file should have about 100M lines.
> But the pfgw.log file stops growing at all after about 70M lines,
although PFGW continues to run.
>
> Furthermore, on killing the task, the file pfgw.log is in a very
bad state.
> Attempting to delete it with
> del pfgw.log
> at the command prompt hangs the operating system.
>
> There is no problem with the same job on files 1/10 the size.
>
> PFGW version:-
> The executable file pfgw.exe is date-stamped 30 Jan 2005, and
says "Version 1.2.0 for Windows".
>
> Operating system:-
> Windows 2000 Professional.
>
> Can anyone shed light on this - and even better correct the
problem?
> Thanks.
>
> -MIke Oakes
>
uses a 32 bit max sized file. I would assume that 70m values in the
pfgw.log file pushed over a 4GB level.
Your only alternative, is to cut the file up.
Jim.
--- In openpfgw@yahoogroup
>
> I believe there is a problem with writing exceptionally long
pfgw.log files.
>
> I am using the command line
> pfgw -b107 xxx.inp
> with an input file coming from NewPGen, and starting:-
> 33319550988:
> 2 71
> 3 71
> ...
> This file is 4 Gb in size, and has 180,259,222 lines.
>
> The pfgw.log file should have about 100M lines.
> But the pfgw.log file stops growing at all after about 70M lines,
although PFGW continues to run.
>
> Furthermore, on killing the task, the file pfgw.log is in a very
bad state.
> Attempting to delete it with
> del pfgw.log
> at the command prompt hangs the operating system.
>
> There is no problem with the same job on files 1/10 the size.
>
> PFGW version:-
> The executable file pfgw.exe is date-stamped 30 Jan 2005, and
says "Version 1.2.0 for Windows".
>
> Operating system:-
> Windows 2000 Professional.
>
> Can anyone shed light on this - and even better correct the
problem?
> Thanks.
>
> -MIke Oakes
>
Offline 
Send Email