Hello,

I'm trying to understand what are reasonable numbers to use when running
dkftpbench.

The weppage says that by default dkftpbench's target bandwidth per
client is set at 28.8 kilobits/second. Is this still a reasonable
assumption to make today ?

I'm running dkftpbench against an ftp server connected through a
cross-over cable. The goal is to measure ftp throughput before and after
a particular automated instrumentation to the source code. dkftpbench
under this configuration always manages to reach the limit of 512
Platoon_MAXUSERS, as set in Platoon.h. However, I see through network
and CPU monitoring tools that the CPU of the server is not saturated yet
under this configuration. Is there an easy way to increase this limit ?
Simply raising the value of this variable in the source code to support
more users reports segmentation faults of dkftpbench.

Looking over other messages in this mailing list I see people reporting
generating load in the order of thousands of requests[1]. Is there
something I'm missing ?

Thanks for any help,
Kristis

[1] - http://tech.groups.yahoo.com/group/ftpbench/message/278



[Non-text portions of this message have been removed]

(Wow, that was fast)

I have a separate test to measure latency of the instrumentation (just
fetch files with wget). My goal here is to measure throughput and see if
the instrumentation I added to vsFTPd v2.0.5 has an impact on
throughput. I assumed that testing with large files would give me this
kind of measure.

In any case, I'm testing with 512 users and a 10K file. I always get a
report from dkftpbench that 510 (not 512) users are left standing. I see
a report at the end like:

ftp_client_pipe_t::notifyPollEvent_cfd: connect failed, errno 24
robouser509::ftpCmdDone: xerr 24, aborting
ftp_client_pipe_t::shutdown(): m_state 1, cfd 1018, dfd -1, id 509
511 users alive, 1 users dead; at least 20 seconds to end of test
ftp_client_pipe_t::notifyPollEvent_cfd: connect failed, errno 24
robouser511::ftpCmdDone: xerr 24, aborting
ftp_client_pipe_t::shutdown(): m_state 1, cfd 1022, dfd -1, id 511
510 users alive, 2 users dead; at least 20 seconds to end of test

If I test with a 32 byte file I can get dkftpbench to report something
510 users alive, like 100-132 users still alive. But I see that the CPU
and network monitoring utilities report utilization is not at 100%.

From what I understand, dkftpbench stops users that die. e.g. if 100 are
still alive, the remaining 412 will not run again. Is this correct ?
Doesn't this leave the server running at less than 100% load ? Or is
perhaps this necessary because otherwise the server crawls to a halt ?

The server machine is an Intel(R) Xeon(TM) CPU 2.40GHz
that reports 4788.23 bogomips under Linux. The client is a Intel(R)
Pentium(R) M processor 1300MHz that reports 2601.87 bogomips under
Linux.

On Mon, 2009-04-13 at 17:59 -0700, Dan Kegel wrote:
> Hi Kristis,
> the many-low-bandwidth-user case was chosen because it
> is exceptionally stressful for the server.  It's still realistic-ish,
> though you might want to also test with few-high-bandwidth-users.
>
> Are you running with a faster client machine than server machine?
>
> It's been a while since I played with ftpbench.
> - Dan
>
> On Mon, Apr 13, 2009 at 5:56 PM, Kristis Makris <kristis.makris@...>
wrote:
> > Hello,
> >
> > I'm trying to understand what are reasonable numbers to use when running
> > dkftpbench.
> >
> > The page says that by default dkftpbench's target bandwidth per client
> > is set at 28.8 kilobits/second. Is this still a reasonable assumption to
> > make today ?
> >
> > I'm running dkftpbench against an ftp server connected through a
> > cross-over cable. The goal is to measure ftp throughput before and after
> > a particular automated instrumentation to the source code. dkftpbench
> > under this configuration always manages to reach the limit of 512
> > Platoon_MAXUSERS, as set in Platoon.h. However, I see through network
> > and CPU monitoring tools that the CPU of the server is not saturated yet
> > under this configuration. Is there an easy way to increase this limit ?
> > Simply raising the value of this variable in the source code to support
> > more users reports segmentation faults of dkftpbench.
> >
> > Looking over other messages in this mailing list I see people reporting
> > generating load in the order of thousands of requests[1]. Is there
> > something I'm missing ?
> >
> > Thanks for any help,
> > Kristis
> >
> > [1] - http://tech.groups.yahoo.com/group/ftpbench/message/278
> >
> >


[Non-text portions of this message have been removed]

Hi!,
after type  ./configure, when i do make check i get:

make  check-TESTS
make[1]: Entering directory `/home/blaxter/tmp/dkftpbench-0.45'
No tests failed.
PASS: ftp_client_proto_test
Test passed
PASS: nbbio_test
check: 0 != 22 at line 303 file Poller_test.cc
FAIL: Poller_test
Sked_test: No tests failed.
PASS: Sked_test
===================
1 of 4 tests failed
===================
make[1]: *** [check-TESTS] Error 1
make[1]: Leaving directory `/home/blaxter/tmp/dkftpbench-0.45'
make: *** [check-am] Error 2


i try to compile in several distro (debian 3, ubuntu 5.10 and fedora
core 2), in all of them i get the same error :-/

(
if i try a make install, the binaries will be make, but when i execute it:

Using poll()
1 users alive, 0 users dead; at least 15 seconds to end of test
robouser0::ftpCmdDone:GETTING: xerr 0, status 550, aborting
ftp_client_pipe_t::shutdown(): m_state 1, cfd 3, dfd -1, id 0
All users dead.  Test failed.

)

--- In ftpbench@yahoogroups.com, Dan Kegel <dank@k...> wrote:
> chetan_the_gaul wrote:
> >  Have started using this tool since a couple of days.
> > Can anyone suggest me the best way to use this to generate load.
> > I have been successful in generating load in the order of thousands of
> > requests against the ftp server i use.
> > But when i have a ftp proxy in between the number of requests does not
> > scale beyond 200. The average seems to be around 130.
> > This is what my active connections show when i check the stats in the
> > proxy.
> > I am looking for a load in the order of thousands in this case too.
>
> Could it be your proxy has a bottleneck?
> - Dan
>

i dont think the proxy is the bottleneck, because the cpu usage and
mem usage is still low in the proxy.
for http load generation with the proxy in place we can scale to more
than 1000 requests. used a different tool for http though. So i am
still wondering where the problem lies.
this is what i did to generate the load.
./dkftpbench -hhost -uuser -ppasswd -fx1000k.dat -n500

-chetan


> --
> Trying to get a job as a c++ developer?  See
http://kegel.com/academy/getting-hired.html

chetan_the_gaul wrote:
>  Have started using this tool since a couple of days.
> Can anyone suggest me the best way to use this to generate load.
> I have been successful in generating load in the order of thousands of
> requests against the ftp server i use.
> But when i have a ftp proxy in between the number of requests does not
> scale beyond 200. The average seems to be around 130.
> This is what my active connections show when i check the stats in the
> proxy.
> I am looking for a load in the order of thousands in this case too.

Could it be your proxy has a bottleneck?
- Dan

--
Trying to get a job as a c++ developer?  See
http://kegel.com/academy/getting-hired.html

Hi,

  Have started using this tool since a couple of days.
Can anyone suggest me the best way to use this to generate load.
I have been successful in generating load in the order of thousands of
requests against the ftp server i use.
But when i have a ftp proxy in between the number of requests does not
scale beyond 200. The average seems to be around 130.
This is what my active connections show when i check the stats in the
proxy.
I am looking for a load in the order of thousands in this case too.
Appreciate if anyone suggests me a way out.

TIA

warm regards,
Chetan

> -----Original Message-----
> From: Dan Kegel [mailto:dank@...]
>
> SIGRTMIN is part of the Real Time Signals optional section of POSIX.
> It would be nice if dkftpbench did not require Real Time Signals to
> compile.
>
> Sounds like Poller_test.cc should conditionalize this line
>                  CHECK(0, m_p.setSignum(SIGRTMIN));
> with #ifdef SIGRTMIN.
>
> As for your second error:
>
>  > Poller_test.cc:251: aggregate `struct sockaddr_in sin' has incomplete
> type
>  > and cannot be initialized ...
>
> Poller_test.cc has the lines
>
> #include <sys/socket.h>
> #include <arpa/inet.h>
> #include <sys/socket.h>
>
> Hmm, maybe one of those should be sys/types.h instead?  Anyway,
> some .h file is missing.  Should be easy to fix.  Let us know what
> it is.
> - Dan

Thanks Dan :)
Here is a patch. (Use with caution!)

%cat patch
--- Poller_test.cc.old  Wed Jul 31 07:59:52 2002
+++ Poller_test.cc      Sun Oct 31 23:16:49 2004
@@ -36,6 +36,10 @@
  #include <arpa/inet.h>
  #include <sys/socket.h>

+/* Added to compile on FreeBSD.
+Also conditionalize statements using SIGRTMIN */
+#include <netinet/in.h>
+
  /*----------------------------------------------------------------------
   Portable function to set a socket into nonblocking mode.
  ----------------------------------------------------------------------*/
@@ -136,7 +140,9 @@
         void testMondo()
         {
                 CHECK(0, m_p.init());
+               #ifdef SIGRTMIN         /* FreeBSD does not have this
defined */
                 CHECK(0, m_p.setSignum(SIGRTMIN));
+               #endif

                 clear();

@@ -238,7 +244,9 @@
         void testRejection(const char *theHost, int thePort)
         {
                 CHECK(0, m_p.init());
+               #ifdef SIGRTMIN
                 CHECK(0, m_p.setSignum(SIGRTMIN));
+               #endif

                 DPRINT(("Connect to a port that will refuse connections,
verify error reported.\n"));
                 // Requires theHost = a nonlocal IP adr, thePort = port
where that host will refuse a connection
@@ -309,7 +317,9 @@
         void testWakeUp()
         {
                 CHECK(0, m_p.init());
+               #ifdef SIGRTMIN
                 CHECK(0, m_p.setSignum(SIGRTMIN));
+               #endif

                 DPRINT(("testWakeUp: Set up a wakeup pipe\n"));
                 CHECK(0, m_p.initWakeUpPipe());
@@ -337,7 +347,9 @@
         {
                 DPRINT(("testCaching\n"));
                 CHECK(0, m_p.init());
+               #ifdef SIGRTMIN
                 CHECK(0, m_p.setSignum(SIGRTMIN));
+               #endif

                 // Create a pipe so we have file descriptors to watch with
Poller.
                 // We use a pipe rather than a file or socket because it's
easy to
%


I know it compiles without a warning. Now to see if it works ;-)


- Nirav Jasapara.
http://www-scf.usc.edu/~jasapara/

"A professional is just an amateur who didn't quit" - Richard Bach.

Nirav Jasapara wrote:
> I am not able to built the sources. I run ./configure and then run make all.
> I get the following error.
>
> FreeBSD ted.isi.edu 4.10-STABLE FreeBSD 4.10-STABLE ...
>
> g++ -DPACKAGE_NAME=\"\" -DPACKAGE_TARNAME=\"\" -DPACKAGE_VERSION=\"\"
> -DPACKAGE_STRING=\"\" -DPACKAGE_BUGREPORT=\"\" -DPACKAGE=\"dkftpbench\"
> -DVERSION=\"0.45\" -DHAVE_INET_ATON=1 -DHAVE_KQUEUE=1 -DHAVE_GETIFADDRS=1
> -DHAVE_DEVPOLL=0 -DHAVE_DEVEPOLL=0 -D_GNU_SOURCE=1 -DSTDC_HEADERS=1
> -DHAVE_SYS_TYPES_H=1 -DHAVE_SYS_STAT_H=1 -DHAVE_STDLIB_H=1 -DHAVE_STRING_H=1
> -DHAVE_MEMORY_H=1 -DHAVE_STRINGS_H=1 -DHAVE_INTTYPES_H=1 -DHAVE_UNISTD_H=1
> -DHAVE_PTHREAD_H=1 -D__x86__=1 -D__freebsd__=1  -I. -I.     -Wall -W
> -Wmissing-prototypes -Wstrict-prototypes -Wmissing-declarations
> -Wwrite-strings -O2 -pthread -c Poller_test.cc  ...
>
> Poller_test.cc:312: `SIGRTMIN' undeclared (first use this function)  ...
>
> Poller_test.cc:251: aggregate `struct sockaddr_in sin' has incomplete type
> and cannot be initialized  ...
>
> Stop in /auto/darkstar/guest/jasapara/ftp/dkftpbench-0.45.
>
> I found the patch sent by Francis for FreeBSD. I see that the tarball has
> already been patched.
>
> I run grep to find 'SIGRTMIN' but I cant find it in any of the headers
> provided. What am I missing ?

SIGRTMIN is part of the Real Time Signals optional section of POSIX.
It would be nice if dkftpbench did not require Real Time Signals to compile.

Sounds like Poller_test.cc should conditionalize this line
                  CHECK(0, m_p.setSignum(SIGRTMIN));
with #ifdef SIGRTMIN.

As for your second error:

  > Poller_test.cc:251: aggregate `struct sockaddr_in sin' has incomplete type
  > and cannot be initialized ...

Poller_test.cc has the lines

#include <sys/socket.h>
#include <arpa/inet.h>
#include <sys/socket.h>

Hmm, maybe one of those should be sys/types.h instead?  Anyway,
some .h file is missing.  Should be easy to fix.  Let us know what
it is.
- Dan

--
Trying to get a job as a c++ developer?  See
http://kegel.com/academy/getting-hired.html

Hi,



I am not able to built the sources. I run ./configure and then run make all.
I get the following error.



------------------

%uname -a

FreeBSD ted.isi.edu 4.10-STABLE FreeBSD 4.10-STABLE #9: Fri Oct 29 17:01:26
PDT 2004     root@...:/usr/obj/usr/src/sys/GENERIC  i386

%

%make all

g++ -DPACKAGE_NAME=\"\" -DPACKAGE_TARNAME=\"\" -DPACKAGE_VERSION=\"\"
-DPACKAGE_STRING=\"\" -DPACKAGE_BUGREPORT=\"\" -DPACKAGE=\"dkftpbench\"
-DVERSION=\"0.45\" -DHAVE_INET_ATON=1 -DHAVE_KQUEUE=1 -DHAVE_GETIFADDRS=1
-DHAVE_DEVPOLL=0 -DHAVE_DEVEPOLL=0 -D_GNU_SOURCE=1 -DSTDC_HEADERS=1
-DHAVE_SYS_TYPES_H=1 -DHAVE_SYS_STAT_H=1 -DHAVE_STDLIB_H=1 -DHAVE_STRING_H=1
-DHAVE_MEMORY_H=1 -DHAVE_STRINGS_H=1 -DHAVE_INTTYPES_H=1 -DHAVE_UNISTD_H=1
-DHAVE_PTHREAD_H=1 -D__x86__=1 -D__freebsd__=1  -I. -I.     -Wall -W
-Wmissing-prototypes -Wstrict-prototypes -Wmissing-declarations
-Wwrite-strings -O2 -pthread -c Poller_test.cc

Poller_test.cc: In method `void Poller_test<Poller_poll>::testWakeUp()':

Poller_test.cc:126:   instantiated from
`Poller_test<Poller_poll>::unitTest()'

Poller_test.cc:397:   instantiated from here

Poller_test.cc:312: `SIGRTMIN' undeclared (first use this function)

Poller_test.cc:312: (Each undeclared identifier is reported only once

Poller_test.cc:312: for each function it appears in.)

Poller_test.cc: In method `void
Poller_test<Poller_poll>::testRejection(const char *, int)':

Poller_test.cc:128:   instantiated from
`Poller_test<Poller_poll>::unitTest()'

Poller_test.cc:397:   instantiated from here

Poller_test.cc:251: aggregate `struct sockaddr_in sin' has incomplete type
and cannot be initialized

Poller_test.cc:257: confused by earlier errors, bailing out

*** Error code 1



Stop in /auto/darkstar/guest/jasapara/ftp/dkftpbench-0.45.

--------------



I found the patch sent by Francis for FreeBSD. I see that the tarball has
already been patched.

I run grep to find 'SIGRTMIN' but I cant find it in any of the headers
provided. What am I missing ?



Thanks in advance.

- Nirav Jasapara.





[Non-text portions of this message have been removed]

Hello Dan!

I hope you still looking up this group!
I searched in the messages for an answer to my question, but didnot
found something new.

I installed dkftpbench on Suse Linux 7.2 (i686) kernel 2.4.26-z1

At first I had a compiling problem. The error message was:

>Platoon.cc: In menber functio 'void Platoon::countStateChange(int,
>            robouser_t::state_t,robuser_t::state_t)':
>Platoon.cc:14: error: 'assert' undeclared (first use in this
>               function)
> Platoon.cc:14: error: (Each undeclared identifier is reported only
>                        once for each function it appears in.)
>Platoon.cc In member function 'void Platoon::reset()':
>Platoon.cc:106: error : 'assert' undeclared (first use in this
>                       function)
>make: ***[Platoon.o] Error 1

To solve this Problem I add
#include <assert.h>
into Platton.cc .

Now I could install dkftpbench. But everytime I run dkftpbench, I
get this error message:
>/usr/local/bin # ./dkftpbench -n1 -hlocalhost -30 -v
>Option values:
> -hlocalhost
> -P21
> -n1
> -c1
> -k2
> -t30
> -b3600
> -B2700
> -uanonymous
> -probouser@
> -fusenet/rec.juggling/juggling.FAQ.Z
> -m1500
> -v1
> -sp
> -a0
>Using poll()
>1 users alive, 0users dead; at least 30seconds to end of test
>robouser0::ftpCmdDone:GETTING: xerr 0, status 550, abborting
>ftp_client_pipe_t::shutdown(): m_state 1, cfd 3, dfd -1, id 0
>All users dead. Test failed.

I run dkftpbench with pureFTPd, proFTPd and NcFTPd (one at time).
I get the same message at every server.
Hope you can help me.
Best regards.
Marco

Dear Dan!

I thought the failed tests are caused by a wrong compilation.
The programm works fine since I use the '-f' option.

But the compilation error still exists.
Is it right to add the assert header ('<assert.h>') at the top of
the Platoon-file?

Marco

cjackeroo wrote:
> I use redhat enterprise Linux 3.0 AS Update 1
>
> Server:  Celeron II 1G,512MB Memory,40GB 7200RPM Harddisk
> Client:  P4 3.2G, 512MB Memory, 120GB 7200RPM Harddisk
>
> no matter what the ftp server I use, serv-u, vsftpd, ncftpd
>
> The maximum connection is 510.
>
> How can i change the maximum connection?

Have you tried increasing ulimit -n in the script that starts the server
process?
- Dan

--
My technical stuff: http://kegel.com
My politics: see http://www.misleader.org for examples of why I'm for regime
change

I use redhat enterprise Linux 3.0 AS Update 1

Server:  Celeron II 1G,512MB Memory,40GB 7200RPM Harddisk
Client:  P4 3.2G, 512MB Memory, 120GB 7200RPM Harddisk

no matter what the ftp server I use, serv-u, vsftpd, ncftpd

The maximum connection is 510.

How can i change the maximum connection?

Richard Pang wrote:
> Dear Dan,
> How could I remove from ftpbench mail list?

Hmm, looks like I should update the trailer on all messages:

> Community email addresses:
>   Post message: ftpbench@onelist.com
>   Subscribe:    ftpbench-subscribe@onelist.com
>   Unsubscribe:  ftpbench-unsubscribe@onelist.com
>   List owner:   ftpbench-owner@onelist.com
>
> Shortcut URL to this page:
>   http://www.onelist.com/community/ftpbench

The correct unsubscribe info is

"To unsubscribe from this group, send an email to:
ftpbench-unsubscribe@yahoogroups.com"

- Dan

Dear Dan,
How could I remove from ftpbench mail list?
Regards,

Tak

--- Dan Kegel <dank@...> ¼g¹D¡G>
"yairinbal" <yairinbal@...> wrote:
> > I am trying to messure 100 concurrent users:
> >
> > ./dkftpbench -h192.168.41.37 -n100 -c100
> -uftpuser0 -pasdf -f/home/ftpuser0/FtpFile
> >
> > I am running RH8
> >
> > Here is part of the output when DPRINT is enabled:
> > countStateChange: user 97, oldstate 1, counts 0 1
> 99 0 0 0 0 0 ;
> > newstate 2, counts 0 0 100 0 0 0 0 0
> > Sked::delClient: index 1, m_used 101
> > Sked::addClient(0xbffbf740, -318121616)
> > waitForEvents: got 100 events
> > getNextEvent: fd 102 revents 5 j 99 m_rfds 99
> > ftp_client_pipe_t::notifyPollEvent(fd 102, 5):
> state 0
> > ftp_client_pipe_t::notifyPollEvent_cfd(fd 102, 5):
> state 0, cning 1
> > ftp_client_pipe_t::notifyPollEvent_cfd: Connect
> cfd 102 succeeded
> > ftp_client_pipe_t::notifyPollEvent_cfd: fillFrom
> returns 32?
> > ftp_client_pipe_t::notifyPollEvent_cfd: fillFrom
> returns 32?
> > ftp_client_pipe_t::notifyPollEvent: xerr 32
> > ftp_client_pipe_t::notifyPollEvent: adding m_cfd
> 102, events 5
> > setMask(102, 5): new mask 5
> > ftp_client_pipe_t::notifyPollEvent:notify_app:
> status 0, dfd_conn 0,
> > dfd -1, xerr 32
> > robouser97::ftpCmdDone: xerr 32, status 0
> > robouser97::ftpCmdDone: xerr 32, aborting  ...
> >
> > I think that poll() returns that all fd's are
> ready, but read()
> > returns 0 bytes.
>
> It seems to think the connection was closed as soon
> as it
> connected.   To get more detail, edit nbbio.cc and
> change
> the #if 0 to #if 1.  But before you do that:
>
> It's always a good idea to try to find the smallest
> number of users
> that causes a problem.  Have you tried fewer users?
> Does 99 users fail
> in the same way?  Does 98?  etc.
>
> Also, have you followed the suggestions in "System
> Tuning" in the
> dkftpbench doc?
> Finally, you might need to set
> /proc/sys/net/ipv4/tcp_tw_recycle to 1
> for large tests.
> - Dan
>

_________________________________________________________
Do You Yahoo!?
µn°O§K¶Oªº @yahoo.com ¤¤¤å¹q¤l¶l¥ó @ http://chinese.mail.yahoo.com
Get your free @yahoo.com address at http://chinese.mail.yahoo.com

Hi,
I am trying to messure 100 concurrent users:

./dkftpbench -h192.168.41.37 -n100 -c100 -uftpuser0 -pasdf -
f/home/ftpuser0/FtpFile

I am running RH8

Here is part of the output when DPRINT is enabled:

.
.
.
countStateChange: user 97, oldstate 1, counts 0 1 99 0 0 0 0 0 ;
newstate 2, counts 0 0 100 0 0 0 0 0
Sked::delClient: index 1, m_used 101
Sked::addClient(0xbffbf740, -318121616)
waitForEvents: got 100 events
getNextEvent: fd 102 revents 5 j 99 m_rfds 99
ftp_client_pipe_t::notifyPollEvent(fd 102, 5): state 0
ftp_client_pipe_t::notifyPollEvent_cfd(fd 102, 5): state 0, cning 1
ftp_client_pipe_t::notifyPollEvent_cfd: Connect cfd 102 succeeded
ftp_client_pipe_t::notifyPollEvent_cfd: fillFrom returns 32?
ftp_client_pipe_t::notifyPollEvent_cfd: fillFrom returns 32?
ftp_client_pipe_t::notifyPollEvent: xerr 32
ftp_client_pipe_t::notifyPollEvent: adding m_cfd 102, events 5
setMask(102, 5): new mask 5
ftp_client_pipe_t::notifyPollEvent:notify_app: status 0, dfd_conn 0,
dfd -1, xerr 32
robouser97::ftpCmdDone: xerr 32, status 0
robouser97::ftpCmdDone: xerr 32, aborting
robouser97::ftpCmdDone: xerr 32, aborting
.
.
.

I think that poll() returns that all fd's are ready, but read()
returns 0 bytes.

Do you have any ideas?

Thanks,
Yair

"yairinbal" <yairinbal@...> wrote:
> I am trying to messure 100 concurrent users:
>
> ./dkftpbench -h192.168.41.37 -n100 -c100 -uftpuser0 -pasdf
-f/home/ftpuser0/FtpFile
>
> I am running RH8
>
> Here is part of the output when DPRINT is enabled:
> countStateChange: user 97, oldstate 1, counts 0 1 99 0 0 0 0 0 ;
> newstate 2, counts 0 0 100 0 0 0 0 0
> Sked::delClient: index 1, m_used 101
> Sked::addClient(0xbffbf740, -318121616)
> waitForEvents: got 100 events
> getNextEvent: fd 102 revents 5 j 99 m_rfds 99
> ftp_client_pipe_t::notifyPollEvent(fd 102, 5): state 0
> ftp_client_pipe_t::notifyPollEvent_cfd(fd 102, 5): state 0, cning 1
> ftp_client_pipe_t::notifyPollEvent_cfd: Connect cfd 102 succeeded
> ftp_client_pipe_t::notifyPollEvent_cfd: fillFrom returns 32?
> ftp_client_pipe_t::notifyPollEvent_cfd: fillFrom returns 32?
> ftp_client_pipe_t::notifyPollEvent: xerr 32
> ftp_client_pipe_t::notifyPollEvent: adding m_cfd 102, events 5
> setMask(102, 5): new mask 5
> ftp_client_pipe_t::notifyPollEvent:notify_app: status 0, dfd_conn 0,
> dfd -1, xerr 32
> robouser97::ftpCmdDone: xerr 32, status 0
> robouser97::ftpCmdDone: xerr 32, aborting  ...
>
> I think that poll() returns that all fd's are ready, but read()
> returns 0 bytes.

It seems to think the connection was closed as soon as it
connected.   To get more detail, edit nbbio.cc and change
the #if 0 to #if 1.  But before you do that:

It's always a good idea to try to find the smallest number of users
that causes a problem.  Have you tried fewer users?  Does 99 users fail
in the same way?  Does 98?  etc.

Also, have you followed the suggestions in "System Tuning" in the
dkftpbench doc?
Finally, you might need to set /proc/sys/net/ipv4/tcp_tw_recycle to 1
for large tests.
- Dan

bunepivahu8881 wrote:
> This is allow you to review and read review about online dating
>
> http://www.revi...

Sorry about that.  I've set the list to moderate new members,
and kicked out the cretin that posted that link.
- Dan

I ran betaftpd under valgrind and stressed it out with
dkftpbench with over 500 users, then killed dkftpbench.
This caused numerous problems to be reported by valgrind.

There were two root causes:

1. list nodes having uninitialized next or prev pointers.
    This was simple to fix, just had to always initialize them to 0.
    Only alloc_new_conn and alloc_new_ftran needed fixing.

2. parse_command calls command handlers, then unless the handler
    returned 0 to indicate a destroyed connection, it references
    the connection again.  However, many handlers call numeric(),
    which can destroy the connection it's being called on.
    The "solution" is to make numeric return the same sort of value
    that the handlers do, and have the handlers return numeric's
    return value.  More or less.

Not quite sure if the fix I did was correct, but it doesn't
fail like it used to under my little stress test.
The fix is at
    http://kegel.com/dkbetaftpd/betaftpd-numeric.patch

On Fri, Oct 04, 2002 at 01:09:22PM -0000, ftpbench@yahoogroups.com wrote:
> ________________________________________________________________________
>
> Message: 2
>    Date: Thu, 03 Oct 2002 15:54:56 -0400
>    From: Kenneth Eaton <keaton@...>
> Subject: compile error
>
> Upon attempting to compile the program I receive the below output and
> error.  Can anyone help out?

I had the same error with Debian Woody v3.0r1, gcc-2.95 was version
2.95.4-11woody1 and g++-2.95 was version 2.95.4-11woody1

I installed gcc-3.0 and g++3.0 with apt-get install gcc-3.0 g++3.0,
then typed:

make clean
export CC=gcc-3.0
export CXX=g++-3.0
./configure
make

and got a clean compile.

With Debian, I can have both versions 2.95 and 3.0 of gcc/g++ installed,
and switch between them with the export command.

-yoshio

Dear all,
How could I modify it to turn it to a benchmark program for streaming
server?
Tak

Dear all,
How could I modify it to turn it to a benchmark program for streaming
server?
Tak

takwonder wrote:
> Dear all,
> I cannot download source code form the website
> http://www.kegel.com/dkftpbench/?
> Why?

No idea.  http://www.kegel.com/dkftpbench/dkftpbench-0.45.tar.gz
downloads fine for me.

> Could this program run on Mac?

On MacOS/X, probably, yes.
On MacOS9.x or earlier, no.
- Dan

--
Dan Kegel
http://www.kegel.com
http://counter.li.org/cgi-bin/runscript/display-person.cgi?user=78045

Dear all,
I cannot download source code form the website
http://www.kegel.com/dkftpbench/?
Why?
If anyone has source code of the benchmark program, could u send to
my email?
Could this program run on Mac?
Thx
Tak

michael_intinarelli wrote:
> Is there a way to test for max sftp client connections to a server
> that uses the sftp_server subsystem of ssh?

I haven't heard of one, but it probably wouldn't be too
hard to throw one together if none exist yet.
- Dan


--
Dan Kegel
http://www.kegel.com
http://counter.li.org/cgi-bin/runscript/display-person.cgi?user=78045

Greetings All,

Is there a way to test for max sftp client connections to a server
that uses the sftp_server subsystem of ssh?

Micael Intinarelli

John Neffenger wrote:
> I removed that page since I think it's obsolete.  Fortunately, I haven't
> had to go recompile the Kernel or the Glibc library in years.

Yeah, it was getting old, and having it around was confusing.

Now that machines are getting so fast, maybe it's worth
running Volanomark up beyond 6000 clients
( that's where http://www.volano.com/report/network.txt stops
at the moment ).  To do so, you might well need to recompile
the Linux kernel to pick CONFIG_3GB.  So maybe an updated
version of that page is called for after all.

Looking at the Volanomark page, I see that Blackdown wasn't
tested with native threads.  Maybe it's time to try that,
since the world seems to be moving in that direction.

Has anyone gotten Volanomark results for the just-released
Blackdown JVM yet?

> Using the following options with Blackdown 1.3.1-02b-FCS on Red Hat
> Linux 8.0, I got 10,000 active socket connections (with 20,000 threads)
> on a 500-MHz Intel Pentium III with 384 megabytes of RAM:
>
>   /usr/local/j2sdk1.3.1/bin/java -green -Xmx256m -Xss64k ...

Note to readers: he's using a smaller thread stack than the
previous poster (64k vs. 100k).  At 1GB address space limit
(or 2^30 bytes), that's an upper limit of 2^(30-16 = 14) = 16000
or so threads.

> All I had to do was increase the per-process file descriptor limits for
> all the socket connections, by doing the following.
>
> Add to /etc/pam.d/login:
>
>   session    required     /lib/security/pam_limits.so
>
> Add to /etc/security/limits.conf:
>
>   *               soft    nofile          1024
>   *               hard    nofile          10240
>
> Don't forget to add a "ulimit -n 10240" to your startup script.

Note that the above only affects future login sessions,
so you'd need to log out and log back in.

Alternately, you can become root, and temporarily raise the limit:
   $ su
   # ulimit -n 10240
   # su dank
   $ java ...
Handy if you're not sure you want to edit limits.conf just yet.

- Dan

--
Dan Kegel
http://www.kegel.com
http://counter.li.org/cgi-bin/runscript/display-person.cgi?user=78045

Sami Farin wrote:
> hmm... little thinko slipped into the code ;)
> fix attached. ...
  > Poller_sigio.cc:
  >   // Set this fd to emit signals.
  > - if (fcntl(fd, F_GETFL, &flags) < 0) {
  > + if ((flags = fcntl(fd, F_GETFL)) < 0) {

Sigh.  I've been warned about that before.  Very sloppy of me, thanks.
(I'm reattaching your two patches to this email for the benefit of
the list.)

> then another bug.  I got this: ...
> Test over.  2499 users left standing.
>
> in reality every user was dead.
> I _think_ the patch attached fixes this, let me know if I got it all wrong.
> it can probably be written in smaller amount of code lines...
  > bench.cc:
  > - printf("Test over.  %d users left standing.\n", nalive);
  > - return 0;
  > + thePlatoon.getStatus(&nconnecting, &nalive, &ndead);
  > + if ((nalive == 0) && (nconnecting == 0)) {
  > +  printf("All users dead.  Test failed.\n");
  > +  exit(1);
  > + } else {
  > +  printf("Test over.  %d users left standing.\n", nalive);
  > +  exit(0);
  > + }

Hmm.  I should look at that a bit more.  That main loop logic
was rather hastily written.  Thanks for the failure log and patch!

> BTW, I get the same performance with poll and rtsig.
> is it normal?  with both poll and rtsig CPU usage is ~100%
> (checked with cyclesoak).
>
> Celeron-A 366 MHz, 256 MiB RAM, linux-2.4.20aa1, ftp server TUX.
>
> $ ./dkftpbench -h127.0.0.1 -c2450 -n2450 -t60 -uftp -fls-Rl.txt \
>              -b1000000000 -B6000 -sr -v0 -m4096
> ...
> Using Linux rtsignals / F_SETSIG / O_ASYNC
> 2450 users alive, 0 users dead; at least 60 seconds to end of test
> Test over.  2450 users left standing.
> $
>
> ls-Rl.txt is 5 MB.
>
> I have done these tunings:
> net.ipv4.tcp_rmem = 16384 16384 16384
> net.ipv4.tcp_max_orphans = 4096
>
> with the default values I get at max 310 connections/sec because
> kswapd starts behaving badly or omething. ;)
>
> kernel: __alloc_pages: 1-order allocation failed (gfp=0x20/0)
> last message repeated 841 times
> kernel: __alloc_pages: 1-order allocation failed (gfp=0x20/1)
> kernel: TCP: too many of orphaned sockets
> last message repeated 3 times
> kernel: NET: 1371 messages suppressed.
>
> disabling window_scaling, timestamps etc do not make any diff on speed.
>
> it would be interesting to test client+server on different machines
> but I have only one machine and 10 Mbit/s network.

Interesting results.  I don't think I ever pushed dkftpbench that
hard.  How long did the run last?  You might need more than 60
seconds to make sure each connection grabbed at least 10 "packets",
which is when the bandwidth check kicks in.  (Come to think of it,
the status function ought to tell you how many connections have
passed that point, shouldn't it?)

I haven't played with dkftpbench for a while, but in some
other experiments I have observed that sigio has trouble
beating select + fork at least in some scenarios (namely
"all connections hot"), whereas /dev/epoll is always a win.
It's about durn time I added sys_epoll support to dkftpbench...

Nakamiin (as my grandma always used to say!)
- Dan

--
Dan Kegel
Linux User #78045
http://www.kegel.com

   ----------

--- dkftpbench-0.45/bench.cc.bak 2002-02-13 07:06:01.000000000 +0200
+++ dkftpbench-0.45/bench.cc 2003-01-06 05:03:21.000000000 +0200
@@ -353,6 +353,12 @@ int main(int argc, char **argv)
 		 }
 	 }
 	 poller->shutdown();
- printf("Test over.  %d users left standing.\n", nalive);
- return 0;
+ thePlatoon.getStatus(&nconnecting, &nalive, &ndead);
+ if ((nalive == 0) && (nconnecting == 0)) {
+  printf("All users dead.  Test failed.\n");
+  exit(1);
+ } else {
+  printf("Test over.  %d users left standing.\n", nalive);
+  exit(0);
+ }
  }

   ----------

--- dkftpbench-0.45/Poller_sigio.cc.bak 2002-07-31 04:26:05.000000000 +0300
+++ dkftpbench-0.45/Poller_sigio.cc 2003-01-06 05:07:32.000000000 +0200
@@ -190,7 +190,7 @@ int Poller_sigio::add(int fd, Client *cl
 	 int err;
 	 int flags;
 	 // Set this fd to emit signals.
- if (fcntl(fd, F_GETFL, &flags) < 0) {
+ if ((flags = fcntl(fd, F_GETFL)) < 0) {
 		 err = errno;
 		 LOG_ERROR(("add: fcntl(fd %d, F_GETFL) returns err %d\n", fd, err));
 		 return err;
@@ -257,7 +257,7 @@ int Poller_sigio::del(int fd)

 	 // Set this fd to no longer emit signals.
 	 int flags;
- if (fcntl(fd, F_GETFL, &flags) < 0) {
+ if ((flags = fcntl(fd, F_GETFL)) < 0) {
 		 err = errno;
 		 LOG_ERROR(("del: fcntl(fd %d, F_GETFL) returns err %d\n", fd, err));
 		 return err;


[Non-text portions of this message have been removed]

Anyone else read this?  It has a very good
high level description of livelock and deadlock in multithreaded
programs in the middle of a very good description of high-performance
server programming.

http://pl.atyp.us/content/tech/servers.html
- Dan

I was shamed into updating my c10k page a bit;
someone at Red Hat commented on how out of date
it was :-(
It now covers recent Linux activity better,
and I've excised some of the dead-end poll alternatives
in favor of the ones that seem to be going forward.
- Dan
http://www.kegel.com/c10k.html