I've been playing with ftpbench and betaftpd on and off,
and sent betaftpd's author a few bugfixes for his daemon.
Haven't had much time to enhance ftpbench lately; I'd like to
get up to 1000 users with *some* daemon before I add too many
more features.

Anyone else playing with it?
- Dan

# of simultaneous users as measured by dkftpbench 0.7:

            wuftpd ncftpd  betaftpd
10kB file   49     194    751
1MB file   120     134    258

See http://www.kegel.com/dkftpbench/results.html

Finally bumped into the 1024 socket limit of select()
with betaftpd on that 751 user run.  Looking forward
to trying the poll() version.

- Dan

After a discussion with Rick Jones (author of netbench),
we both agreed to use "each session must achieve at least
75% of specified bandwidth" as the criterion for success.
(I'd been using 80%; he'd been using 66% of average bw.)
This should bring dkftpbench results more in line with
netbench 3.0 results.

I've also added a timeout of 5 seconds for connect+login
time, and 5 seconds for "time to get first packet".

On the same P90 server / PIII-450 server as before, with the same
100k files, but with 100baseT, I now get 700 users solidly
on betaftpd-pre11.
Oddly, this doesn't violate the 1024 fd limit on betaftpd,
as only 750 or so fd's are open; I guess sendfile finishes
early, and the data sockets get closed and don't count toward
the limit.

Switching to 1 megabyte files, I only get 255 users; the next
user takes more than 5 seconds to get the first packet.
The number of open files is still down around 750.
Haven't looked into why yet.

I haven't tried Steinar's version that uses poll() yet;
took me a while to figure out why old version worked at
700 users.
- Dan

On Tue, 18 Jan 2000, you wrote:
> From: Dan Kegel <dank@...>
>
> 0.6 ramps up more gently, which gets rid of a lot of
> problems I'd been having with betaftpd.
> It's also less verbose by default; to get the old output,
> you have to give one more -v.
>
> betaftp now handles 300 users indefinitely on my P90/32MB machine.
Hi again,

I just want to confirm the results you got for betaftpd, I played with
betaftpd on that 486-133Mhz (36meg ram) machine of mine. It yeilded about 300
users, at 65-75% load on the 486 machine. So it looks like betaftpd could
potentially run away with the cookies :-), if its main select loop is replaced
by a poll loop.

Cya,
Beau Kuiper
ekuiperba@...
Muddleftpd: http://www.arach.net.au/~wildfire/muddleftpd

> (betaftp uses 70% of the CPU with that load, btw.)
> Trying for 320 users usually ends up running 315 or so.
> - Dan
>   http://www.kegel.com/dkftpbench/

On Fri, Jan 21, 2000 at 12:30:34AM +0800 or thereabouts, Beau Kuiper wrote:

> wasn't heavily optimized (and hit the process maximum limit of linux)

fwiw, this is fixed in 2.3.  you can fill pid_t space with running jobs,
from what I recall.

--
- zach

Switched to 100baseT (Netgear FA310x, $25, uses Tulip driver).
Ran this script on both client and server (using '. tune.sh'):
  ulimit -n 4096
  echo 1024 32767 > /proc/sys/net/ipv4/ip_local_port_range
  echo 4096 > /proc/sys/fs/file-max
just to be sure I wasn't going to run out of sockets.

I can now do 500 users with betaftpd0.0.8pre11 indefinitely.
CPU usage is down to 50% at that load, thanks to the new cards.
Nearly all of the CPU load for both betaftpd and dkftpbench is
system time; maybe 1/6 is user time.

At that number of users, dkftpbench was very low on RAM at 64MB, and
betaftpd was very low on RAM at 32MB.  Might need more RAM on both
to go much higher.

Noticed a few bugs:
1) betaftpd's check for too-many-users is broken, and it accesses
things off the end of the fdset.  Should range-check the fd's.
2) betaftpd doesn't notice if a simulated user never gets a single
packet.  It should time out.  This bug caused it to think betaftpd
was working fine at 600 users.

- Dan

On Fri, 21 Jan 2000, you wrote:
> From: Dan Kegel <dank@...>
>
> Beau Kuiper wrote:
> > > 0.6 ramps up more gently, which gets rid of a lot of
> > > problems I'd been having with betaftpd. ...
> > > betaftp now handles 300 users indefinitely on my P90/32MB machine.
> > > (betaftp uses 70% of the CPU with that load, btw.)
> >
> > Yep, In my own testing, the new version of muddleftpd I am working on ...
> > ramps up to about 260 users indefinitely on a 486-133mhz with 36 meg
> > ram ... The slower ramp up code makes a lot of difference. (note muddleftpd
> > is a process per connection design, like proftpd or wu-ftpd, only much
faster ...
>
> Thanks for the feedback.  Nice to have more numbers to compare with!
Cool ;-)
>
> > Basicly, most ftp servers can
> > saturate a 10mbps network on such a simple workload. Correct me if I am
wrong,
> > but 260 clients doing real work (not all cached data, very large files,
listing)
> > would leave a 486 inoperable, even running betaftpd :-)
>
> Yep.  Feel free to report results on fetching 100 megabyte files, though :-)

I feel what is really needed is lots of files that the server chooses (at
random, or in a sequence), so that disk accesses occurs.

>
> > I am wondering now, is listing going to be tested at all in the benchmark. A
> > significant amount of work went into most servers optimising this rather
than
> > the simple case binary file download (just how much faster can you get). I
> > imagine ncftpd would whip muddleftpd in list performace, while server
> > performace would suffer slightly on betaftpd.
>
> Yes, I plan to add directory listing to the workload.

Ah cool :-)

>
> The workload is so simple right now because it's enough work doing even
> that right, and I didn't want to complicate matters until I could get
> several thousand users running with it.  I've still only seen 300 users
> myself, so I have another order of magnitude to go before I start improving
> the workload.  (I have 100baseT cards now, so maybe soon...)
> - Dan

I have occasional access to reasonably serious hardware with 100mbps cards,
128meg ram, SCSI hardware, P-II CPU. I think I got to about 450 users with an
earlier version of dkftpbench using such a machine as a server. The server
wasn't heavily optimized (and hit the process maximum limit of linux)

Have Fun
Beau Kuiper
ekuiperba@...

Beau Kuiper wrote:
> > 0.6 ramps up more gently, which gets rid of a lot of
> > problems I'd been having with betaftpd. ...
> > betaftp now handles 300 users indefinitely on my P90/32MB machine.
> > (betaftp uses 70% of the CPU with that load, btw.)
>
> Yep, In my own testing, the new version of muddleftpd I am working on ...
> ramps up to about 260 users indefinitely on a 486-133mhz with 36 meg
> ram ... The slower ramp up code makes a lot of difference. (note muddleftpd
> is a process per connection design, like proftpd or wu-ftpd, only much faster
...

Thanks for the feedback.  Nice to have more numbers to compare with!

> Basicly, most ftp servers can
> saturate a 10mbps network on such a simple workload. Correct me if I am wrong,
> but 260 clients doing real work (not all cached data, very large files,
listing)
> would leave a 486 inoperable, even running betaftpd :-)

Yep.  Feel free to report results on fetching 100 megabyte files, though :-)

> I am wondering now, is listing going to be tested at all in the benchmark. A
> significant amount of work went into most servers optimising this rather than
> the simple case binary file download (just how much faster can you get). I
> imagine ncftpd would whip muddleftpd in list performace, while server
> performace would suffer slightly on betaftpd.

Yes, I plan to add directory listing to the workload.

The workload is so simple right now because it's enough work doing even
that right, and I didn't want to complicate matters until I could get
several thousand users running with it.  I've still only seen 300 users
myself, so I have another order of magnitude to go before I start improving
the workload.  (I have 100baseT cards now, so maybe soon...)
- Dan

Hi,

On Tue, 18 Jan 2000, you wrote:
> From: Dan Kegel <dank@...>
>
> 0.6 ramps up more gently, which gets rid of a lot of
> problems I'd been having with betaftpd.
> It's also less verbose by default; to get the old output,
> you have to give one more -v.
>
> betaftp now handles 300 users indefinitely on my P90/32MB machine.
> (betaftp uses 70% of the CPU with that load, btw.)
> Trying for 320 users usually ends up running 315 or so.

Yep, In my own testing, the new version of muddleftpd I am working on (should be
ready for release soon) ramps up to about 260 users indefinitely on a 486-133mhz
with 36 meg ram (not much optimization). The slower ramp up code makes a lot of
difference. (note muddleftpd is a process per connection design, like proftpd
or wu-ftpd, only much faster than both :-). ). Basicly, most ftp servers can
saturate a 10mbps network on such a simple workload. Correct me if I am wrong,
but 260 clients doing real work (not all cached data, very large files, listing)
would leave a 486 inoperable, even running betaftpd :-)

I am wondering now, is listing going to be tested at all in the benchmark. A
significant amount of work went into most servers optimising this rather than
the simple case binary file download (just how much faster can you get). I
imagine ncftpd would whip muddleftpd in list performace, while server
performace would suffer slightly on betaftpd.

Anyway, I will quit my ramblings, Have fun
Beau Kuiper
ekuiperba@...


> - Dan
>
> --------------------------- ONElist Sponsor ----------------------------
>
> Independent contractors: Find your next project gig through JobSwarm!
>         You can even make money by referring friends.
> <a href=" http://clickme.onelist.com/ad/jobswarm2 ">Click Here</a>
>
> ------------------------------------------------------------------------
>
> Community email addresses:
>   Post message: ftpbench@onelist.com
>   Subscribe:    ftpbench-subscribe@onelist.com
>   Unsubscribe:  ftpbench-unsubscribe@onelist.com
>   List owner:   ftpbench-owner@onelist.com
>
> Shortcut URL to this page:
>   http://www.onelist.com/community/ftpbench
>
> Code:
>   http://www.kegel.com/dkftpbench/
-------------------------------------------------------

0.6 ramps up more gently, which gets rid of a lot of
problems I'd been having with betaftpd.
It's also less verbose by default; to get the old output,
you have to give one more -v.

betaftp now handles 300 users indefinitely on my P90/32MB machine.
(betaftp uses 70% of the CPU with that load, btw.)
Trying for 320 users usually ends up running 315 or so.
- Dan

I've posted a comparison of wu-ftpd and betaftpd at
   http://www.kegel.com/dkftpbench/results.html
on a very simple benchmark (fetching the same 100kilobyte
file over and over again, limited to 28000 bits/sec on
the receiving end).

betaftpd handled 249 users at about 28000 bits/sec each, for an
average throughput of something like 6 megabits/sec.
Not bad for a 10baseT connection.

wu-ftpd pooped out at 39 users.

Your milage may vary, especially if you are serving a
large real document tree instead of a single 100 kilobyte file,
or are running over a real WAN instead of my cheesy simulation.

- Dan

As a load generator, it's getting pretty good; you can now tell
whether connection problems are the client's fault (it aborts if
it notices any client-side resource shortage) or the server's
fault (in which case it prints out more exactly what happens).
Still a little cryptic, but useful nonetheless.

Thanks to Steinar Gunderson (author of betaftpd) for all the feedback.
- Dan

CHANGELOG for dkftpbench

0.4: 13 Jan 1999
         Includes tuning utility 'dklimits'.
         More verbose error messages when a client connection dies.
         Abort benchmark if running out of local resources.

0.3: 4 Jan 1999
         Now prints out useful message when killing user due to slowness.
         Now checks continuously for sufficient bandwidth, not just at end of
fetch.
         This makes bench more cautious about creating new users, and
         more demanding of continuous high performance before it agrees the
server
         can support a given # of users.

         The callbacks are getting tricky...
         ftp_client_pipe::handle_io now returns EPIPE if handle_data_io returns
-1
         ftp_client_pipe::skedCallback now calls shutdown if handle_io returns
error
         robouser::unwatchfd now calls stop if cfd is true
         robouser::stop now defers calling shutdown until static_reap to avoid
recursion
         ftp_client_pipe now slightly more paranoid about recursion when calling
unwatchfd

         Fixed bug in Sked::remove() which showed up when ftp server killed
during run
0.2: 3 Jan 1999
         Added 'make data'
         Added interim reporting guidelines in index.html
         No longer abort upon delayed connection refusal
         Check for data connection completion at proper time
         Works better with BetaFTPd now

0.1: 2 Jan 1999
         Initial release

Rick Jones warned me that the traffic generated by
dkftpbench may trigger problems in some tcp stacks.
See http://www.deja.com/getdoc.xp?AN=568757304

I have tested so far only with betaftpd, and am
seeing some anomalies that may be TCP stack problems.
Before I jump to that conclusion, I will try to
reproduce the problems using ncftpd, which is
commercially supported and should be stable
(betaftpd is still an unstable development server).

Although most applications won't tickle these problems,
it's not unimaginable that backing up a fast
disk to a slow tape over the network would
produce similar traffic.
This may be an opportunity to expose and clean
up obscure problems in tcp stacks.

- Dan

This fixes a couple bugs without introducing any, I hope.

It now kills simulated users as soon as their average bandwidth
drops below 80% of the target, rather than waiting 'til the
end of the fetch.  This is pickier, but avoids lame duck users
dragging everyone else down, and avoids creating scads of users
when things are already falling apart.  These problems were more
noticable on the 100k file than the 10k file.

Running on my P133 laptop against a P90 running betaftpd, all under RH6.0,
I get between 45 and 65 users with the 100 kbyte file,
and 60-65 users with the 10k file.

- Dan

p.s. If you downloaded 0.3 already, download it again...

--
(The above is just my personal opinion; I don't speak for my employer,
  except on the occasional talk show.)

Implemented throttling over the holidays (mostly while
on airplanes...) and cleaned up the code a bit.
It's at http://www.kegel.com/dkftpbench/ and works fairly well.
Lots to do, still, but at least it's alive...
guess I'll mention it at freshmeat.net.
- Dan

--
(The above is just my personal opinion; I don't speak for my employer,
  except on the occasional talk show.)

Trevor Johnson wrote:
> It's a common practice to tar up the whole directory.  It makes things a
> tiny bit easier for users (especially if they forget to test the tarball
> first, and extract it in a directory that has other stuff).
OK, I've switched to follow the package-major.minor naming convention
for both the tarball itself and the directory it contains.

> My g++ (version 2.7.2.3) choked on the -Wsign-compare option
OK, removed.

> Here are the results I got on a PC with a 133 MHz (PR-166) 6x86 CPU and 32
> MB of RAM, in idle multi-user mode:
>...
>In file included from Sked.cc:41:
>/usr/include/values.h:2: warning: #warning "this file includes <values.h> which
>is obsoleted, use <limits.h> or <float.h> instead"
Fixed.

>/usr/include/signal.h:83: warning: `int sigvec(int, struct sigvec *, struct
>sigvec *)' hides constructor for `struct sigvec'
Presumed benign.  I'll leave that alone (not much I could do anyway).

>t100k 2.781250; t10k 0.218750; ratio 12.714286; time to do 1 at 10k: 0.000022
>No tests failed.
>10:~/src/ftpbench$ uname -a
>FreeBSD 10.0.0.99 3.4-RC FreeBSD 3.4-RC #0: Sat Dec 11 20:28:09 PST 1999
trevor@...:/usr/src/sys/compile/FREEBASE  i386

Cool, it works!  Thanks for the help.

> I forgot to mention:  a good place to do portability testing is Compaq's
> test-drive site.  They have FreeBSD 3.3 and 4.0, Linux 2.2, Tru64 and
> Open[sic]VMS on PCs and Alpha systems. The details are at
> http://www.testdrive.compaq.com

Thanks for the tip.  I'll sign up there.

The sources at http://www.kegel.com/ftpbench/ are updated with your
suggested changes.  I appreciate your help.

I suppose it's a little odd to work on portability before the whole
package is done, but since I'm doing the portability thing for my real
job at the moment, it was easy to do it to this project, too.
Besides, it helped flush out a bug or two.
I hope to have benchmark results in time for April Fools' Day :-)
- Dan

--
(The above is just my personal opinion; I don't speak for my employer,
  except on the occasional talk show.)

'make all' works and passes self-tests on Solaris now,
even gunky old solaris 2.6 :-)

Turned out I was writing to the socket before the connect
had finished.  Oops.
- Dan

--
(The above is just my personal opinion; I don't speak for my employer,
  except on the occasional talk show.)

I've switched over to using autoconf to handle
system dependencies instead of trying to write
manual checks.

README explains how to build; it's now
   ./configure
   make all

Look at configure.in if you're curious about autoconf.
It's a hard tool to use, i.e. it's hard to write
configure.in files, but it's worth it because it makes it
easier to achieve portability across various flavors of Unix.
- Dan

--
(The above is just my personal opinion; I don't speak for my employer,
  except on the occasional talk show.)

When robousers are deleted, it's important to delete their
outstanding events from the scheduler's priority queue,
so today I added a delClient() method to Sked.

Along the way, I rewrote Sked::isHeap; Mark wrote it
recursively, but "Programming Pearls" had an iterative
version that was much simpler, so I switched to that.
- Dan
--
(The above is just my personal opinion; I don't speak for my employer,
  except on the occasional talk show.)

Today I spent some time on making the ftpbench code compile
on Solaris and FreeBSD (we already had one fellow send in a
patch for FreeBSD last month).
Cleaned up a lot of minor things, but it fails an assertion
or two on Solaris.  Guess I'll look at those tomorrow.

Man do I love unit tests!  Without those little sanity checks,
porting to other platforms would be more painful.

Also, for what it's worth, porting to another OS often exposes
bugs that had been hiding on the original OS, waiting for
an inconvenient moment to come out and bite you.  It's good
to port.
- Dan

--
(The above is just my personal opinion; I don't speak for my employer,
  except on the occasional talk show.)

On Sat, 18 Dec 1999, Dan Kegel wrote:

> If you do cat first, the read will block until there's data.
> If you do echo first, the write will block until there's a reader...

Cool...  I get it now. :-)

bb wrote:
> > Good question.  You need setNonblocking() because unless you call it,
> > the functions won't return immediately, because they call read().
> > read()'s behavior is changed by setNonblocking() from the default
> > (blocking).
>
> I still don't get this.  The code looks like it is forcing only one
> buf_len to be read, sent, and then the function returns.  How can read not
> return immediately?  What else is it doing?  I read man read(),  and there
> is nothing about blocking.  Do you mean that it blocks any other read()
> from executing?

read() does indeed block if it's reading from a socket or a pipe
and it's trying to read more bytes than are there at the moment.
Similarly, write() will block if it's writing to a socket or pipe
and it's trying to write more bytes than there's room for at the moment.
This is called flow control.

You can see flow control in action even from the commandline.
Use 'mknod mypipe p' to create a named pipe, then
do 'cat mypipe' in one window, and 'echo foo > mypipe' in another window.
Try doing the cat first, and then the echo first.
If you do cat first, the read will block until there's data.
If you do echo first, the write will block until there's a reader...
- Dan

--
(The above is just my personal opinion; I don't speak for my employer,
  except on the occasional talk show.)

On Sat, 18 Dec 1999, Dan Kegel wrote:

>
> Good question.  You need setNonblocking() because unless you call it,
> the functions won't return immediately, because they call read().
> read()'s behavior is changed by setNonblocking() from the default
> (blocking).

I still don't get this.  The code looks like it is forcing only one
buf_len to be read, sent, and then the function returns.  How can read not
return immediately?  What else is it doing?  I read man read(),  and there
is nothing about blocking.  Do you mean that it blocks any other read()
from executing?

bb wrote:
> > I've added a page, http://www.kegel.com/ftpbench/theory.html
> Hi Dan.  I have been studying this.  I have a couple of questions.  If the
> functions are written to return immediately, why do you need a function
> like setNonblocking()?

Good question.  You need setNonblocking() because unless you call it,
the functions won't return immediately, because they call read().
read()'s behavior is changed by setNonblocking() from the default
(blocking).

> Also I don't understand this:
>  But in the world of nonblocking programming, you can't do this,
>  because it loops until some external event happens (i.e. the socket
>  accepts the whole file), which is a major no-no.
> Why is it a major no-no?  Is it only bad because you want to send multiple
> files?

Yes, it's only bad because you want to do two things at once,
not fixate on one until it's done.

Let's say you're trying to service more than one remote peer at a time.
If you loop waiting for the first peer to accept all the data you're
sending to it, the second peer will be starved for attention.

Web servers can't very well wait for the first client to accept
all of a big file before sending stuff to their other clients...
- Dan

--
(The above is just my personal opinion; I don't speak for my employer,
  except on the occasional talk show.)

Hi Dan.  I have been studying this.  I have a couple of questions.  If the
functions are written to return immediately, why do you need a function
like setNonblocking()?  Also I don't understand this:
  But in the world of nonblocking programming, you can't do this,
  because it loops until some external event happens (i.e. the socket
  accepts the whole file), which is a major no-no.
Why is it a major no-no?  Is it only bad because you want to send multiple
files?

Barbara

On Sun, 12 Dec 1999, Dan Kegel wrote:

> From: Dan Kegel <dank@...>
>
> I've added a page, http://www.kegel.com/ftpbench/theory.html
> on the theory of operation of the code, including a little
> tutorial on how to get your head around nonblocking code.
> - Dan
>
> --
> (The above is just my personal opinion; I don't speak for my employer,
>  except on the occasional talk show.)
>
> > Community email addresses:
>   Post message: ftpbench@onelist.com
>   Subscribe:    ftpbench-subscribe@onelist.com
>   Unsubscribe:  ftpbench-unsubscribe@onelist.com
>   List owner:   ftpbench-owner@onelist.com
>
> Shortcut URL to this page:
>   http://www.onelist.com/community/ftpbench
>
> Code:
>   http://www.kegel.com/ftpbench/
>

Throttling thousands of client connections requires the
ability to schedule future operations efficiently.
We can't use the operating system's scheduler (since we're
doing a single-threaded program), so
I had Mark Williams implement a scheduler module using an efficient
priority queue algorithm from Jon Bentley's book "Programming Pearls"
(http://www.awl-he.com/titles/13184.html,
  http://store.yahoo.com/softpro/0-201-65788-0.html).
I changed robouser over to use this new Sked module rather
than its built-in scheduler.
The pieces are now in place for me to add throttling
to ftp_client_pipe.
New code and doc uploaded.
- Dan

--
(The above is just my personal opinion; I don't speak for my employer,
  except on the occasional talk show.)

Working on throttling.
The callback now returns how many bytes were fetched.
Added new module Sked.h.  Because ftp_client_pipe will
soon have to schedule future stuff, I'll move the scheduling
out of robouser and into Sked, then make both robouser and ftp_client_pipe
use Sked's services.

Minor changes, including Sked.h, uploaded.
- Dan
--
(The above is just my personal opinion; I don't speak for my employer,
  except on the occasional talk show.)

I've added a page, http://www.kegel.com/ftpbench/theory.html
on the theory of operation of the code, including a little
tutorial on how to get your head around nonblocking code.
- Dan

--
(The above is just my personal opinion; I don't speak for my employer,
  except on the occasional talk show.)

robouser no longer breaks when you fetch small local files;
I cleaned up the GETTING state machine so it makes sense.
Also added an option to specify which file to fetch.

OK, *now* maybe I'll work on throttling next.
- Dan
--
(The above is just my personal opinion; I don't speak for my employer,
  except on the occasional talk show.)

This is the first post to the new mailing list, welcome aboard.

I notice a subscriber or two whose email address I don't recognize-
should we do introductions?

http://www.kegel.com/ftpbench/ now includes a link
to this mailing list, and a Status section with a
to-do list.  Throttling is going to be a bit tricky to add...
maybe I'll tackle that next.

I've updated the code to fix a few buglets, to
print out overall bandwidth once per five seconds,
and to terminate if the bandwidth goes to zero.
index.html no longer mentions clftp.cc, as it's out of date.

If anyone wants to try implementing something interesting, try doing
ftp_client_proto::put() and ftp_client_pipe::put();
that'll be handy for when we want to have bench upload
its own test files to the server.
- Dan

--
(The above is just my personal opinion; I don't speak for my employer,
  except on the occasional talk show.)