Search the web
Sign In
New User? Sign Up
openpfgw · Co-ordination of the OpenPFGW project
? Already a member? Sign in to Yahoo!

Yahoo! Groups Tips

Did you know...
Hear how Yahoo! Groups has changed the lives of others. Take me there.

Best of Y! Groups

   Check them out and nominate your group.
Having problems with message search? Fill out this form to ensure your group is one of the first to be migrated to the new message search system.

Messages

  Messages Help
Advanced
Bug in processing huge input files from NewPGen   Message List  
Reply | Forward Message #1831 of 2072 |
Re: Bug in processing huge input files from NewPGen

OpenPFGW (win32) uses FILE structures, which under VC (and MinGW),
uses a 32 bit max sized file. I would assume that 70m values in the
pfgw.log file pushed over a 4GB level.

Your only alternative, is to cut the file up.

Jim.

--- In openpfgw@yahoogroups.com, mikeoakes2@... wrote:
>
> I believe there is a problem with writing exceptionally long
pfgw.log files.
>
> I am using the command line
> pfgw -b107 xxx.inp
> with an input file coming from NewPGen, and starting:-
> 33319550988:P:0:2:65
> 2 71
> 3 71
> ...
> This file is 4 Gb in size, and has 180,259,222 lines.
>
> The pfgw.log file should have about 100M lines.
> But the pfgw.log file stops growing at all after about 70M lines,
although PFGW continues to run.
>
> Furthermore, on killing the task, the file pfgw.log is in a very
bad state.
> Attempting to delete it with
> del pfgw.log
> at the command prompt hangs the operating system.
>
> There is no problem with the same job on files 1/10 the size.
>
> PFGW version:-
> The executable file pfgw.exe is date-stamped 30 Jan 2005, and
says "Version 1.2.0 for Windows".
>
> Operating system:-
> Windows 2000 Professional.
>
> Can anyone shed light on this - and even better correct the
problem?
> Thanks.
>
> -MIke Oakes
>








Tue Jun 6, 2006 4:09 pm

jim_fougeron
Offline Offline
Send Email Send Email

Forward
Message #1831 of 2072 |
Expand Messages Author Sort by Date

I believe there is a problem with writing exceptionally long pfgw.log files. I am using the command line pfgw -b107 xxx.inp with an input file coming from...
mikeoakes2@...
mikeoakes2
Offline Send Email
Jun 6, 2006
1:22 pm

OpenPFGW (win32) uses FILE structures, which under VC (and MinGW), uses a 32 bit max sized file. I would assume that 70m values in the pfgw.log file pushed...
jim_fougeron
Offline Send Email
Jun 6, 2006
4:11 pm

Thanks Jim. I understand that there are several ways round this. However... This problem came upon me unawares, and could do the same to anyone at any time. A...
mikeoakes2@...
mikeoakes2
Offline Send Email
Jun 6, 2006
7:40 pm

Hello Mike, ... What do you mean by "quarantineing" the files? What kind of filesystem do you use on your disk, by the way (FAT32? NTFS?) ? Have you tried to...
Peter Kosinar
pkosinar
Offline Send Email
Jun 6, 2006
10:45 pm

... The HD file system is NTFS. After a reboot, you can /rename/ the file pfgw.log, but any attempt to delete it causes the command prompt to go to 99% cpu...
mikeoakes2@...
mikeoakes2
Offline Send Email
Jun 7, 2006
8:22 am

... That looks like 2^30 bytes minus just under 1 MB of unflushed cached writes. ... This sounds like a case for backing up your data - writable DVDs, CDRs, ...
Phil Carmody
thefatphil
Offline Send Email
Jun 7, 2006
4:00 pm

... PFGW doesn't write any files. It asks the OS to write data to the filesystem. That's what OSes are for. ... This OS bug. ... Same company - same mindset. ...
Phil Carmody
thefatphil
Offline Send Email
Jun 7, 2006
7:55 am

... From: Phil Carmody <thefatphil@...> To: openpfgw@yahoogroups.com Sent: Wed, 7 Jun 2006 00:53:57 -0700 (PDT) Subject: Re: [openpfgw] Re: Bug in...
mikeoakes2@...
mikeoakes2
Offline Send Email
Jun 7, 2006
9:06 am

... You assume I am presuming. I'm not. I _know_ what an OS's job is, I don't have to make guesses. ... 1 fopen, many fprintfs/fputss, 1 fclose or many {...
Phil Carmody
thefatphil
Offline Send Email
Jun 7, 2006
1:12 pm

This "bug" is most likely compounded due to the fseek in PFGW. However, step back for a second. First off, why are you using PFGW to PRP test a*71#+1 ?? PFGW...
jim_fougeron
Offline Send Email
Jun 7, 2006
2:58 pm

... It's done in a very simple manner (don't have the actual sources by hand; checking somewhat older version but I really doubt it has changed much since...
Peter Kosinar
pkosinar
Offline Send Email
Jun 7, 2006
3:57 pm

hello, I have been following this thread about the log file of pfgw being too large. I use cygwin to run pfgw and I can manipulate files that are larger than...
Simon Plouffe
badaboum0
Offline Send Email
Jun 6, 2006
7:53 pm

... I agree with the conclusion (and of course, move each pfgw.log aside before ... Each pfgw.log line will be 4 characters longer than the above. "a b" ->...
Phil Carmody
thefatphil
Offline Send Email
Jun 7, 2006
7:03 am

No-one seems to have pinpointed code which could explain where misbehaviour at c. 1 Gb o/p file size is coming from. [BTW NewPGen, which Paul J tells me also...
mikeoakes2@...
mikeoakes2
Offline Send Email
Jun 8, 2006
8:54 am

... Paul found it - it's either or both of the "t" in the fopen mode, and/or the unnecessary fseek. As DOS users like their carriage returns, I think even ANSI...
Phil Carmody
thefatphil
Offline Send Email
Jun 8, 2006
11:34 am

One month later, and still no response at all on this pfgw Winxxx problem, or even for what Winxxx variant(s) it is :-( (I don't count responses which says...
mikeoakes2@...
mikeoakes2
Offline Send Email
Jul 10, 2006
8:37 am
Advanced

Copyright © 2009 Yahoo! Inc. All rights reserved.
Privacy Policy - Terms of Service - Guidelines - Help