Subject: [htdig3-dev] Re: Fwd: Re: [htdig] htdig pausing with each new server (PR#670) (fwd)
From: Gilles Detillieux (grdetil@scrc.umanitoba.ca)
Date: Wed Nov 10 1999 - 14:17:35 PST
Oops, I had meant to CC this to htdig3-dev last week. No wonder no one
commented on this. :-P Can anyone shed some light on what the proper
behaviour should be regarding closing connections on errors?
> From: grdetil@scrc.umanitoba.ca
> To: htdig3-bugs@htdig.org
> Message-ID: <382071B6.BeroList-2.5.9@sob.htdig.org>
> Cc: htdig3-bugs@htdig.org
> Date: Wed, 3 Nov 1999 09:32:37 -0800 (PST)
> Subject: Re: Fwd: Re: [htdig] htdig pausing with each new server (PR#670)
>
> In 3.1.3 and earlier, Document::RetrieveHTTP() only does a c.close() at
> the end, before returning Document_ok. If it returns earlier with any
> other return code, it doesn't do the c.close(). Is this the problem?
>
> As for 3.2, I don't understand the new htnet/HtHTTP.cc code enough to
> tell if there's a problem there or not. The persistent connection stuff
> complicates things a lot, but it seems there are some error situations
> that aren't dealt with thoroughly enough.
>
> According to Geoff Hutchison:
> > >He promised to send a netstat report when he does his next dig. Open
> > >connections aren't a great thing, especially since it's slowing down
> > >the servers it's indexing.
> > >
> > >-Geoff
> > >
> > > >From: Steven Karel <karelsf@brandeis.edu>
> > > >X-Sender: karelsf@squirrel.bio.brandeis.edu
> > > >To: Geoff Hutchison <ghutchis@wso.williams.edu>
> > > >Subject: Re: [htdig] htdig pausing with each new server
> > > >
> > > >On Tue, 5 Oct 1999, Geoff Hutchison wrote:
> > > >
> > > > > At 10:00 AM -0400 10/5/99, Steven Karel wrote:
> > > > > >If htdig isn't multithreaded, that's a problem -- I wish I had the time
> > > > > >and expertise to help. But I'm still curious -- if it isn't
> > >multithreaded,
> > > > > >how does it open dozens of simultaneous connections when it's in the
> > > > > >retrieval part of the dig (as opposed to the server
> > >identification part)?
> > > > >
> > > > > It doesn't open dozens of simultaneous connections! It only opens a
> > > > > connection for each file. Granted it generally flies through indexing
> > > > > each file, but there shouldn't be dozens of simultaneous connections.
> > > >
> > > >It does create dozens of simultaneous connections (one for each file,
> > > >but many connections to each server) -- possibly through the
> > > >failure to close the connections it opens? At one point I got yelled at by
> > > >one of the other sysadmins for a denial-of-service attack on his webserver
> > > >-- it had hundreds of open connections. Increasing server-wait-time helped
> > > >with that, but I still find that if I look with netstat while htdig is
> > > >running that there are a large number (dozens) of connections attributable
> > > >to htdig.
-- Gilles R. Detillieux E-mail: <grdetil@scrc.umanitoba.ca> Spinal Cord Research Centre WWW: http://www.scrc.umanitoba.ca/~grdetil Dept. Physiology, U. of Manitoba Phone: (204)789-3766 Winnipeg, MB R3E 3J7 (Canada) Fax: (204)789-3930------------------------------------ To unsubscribe from the htdig3-dev mailing list, send a message to htdig3-dev@htdig.org containing the single word "unsubscribe" in the SUBJECT of the message.
This archive was generated by hypermail 2b25 : Wed Nov 10 1999 - 14:28:07 PST