 |
|
|
|
Next: Accepted tnftp 20090520-1 (source amd64)
|
| Author |
Message |
External
Since: Nov 26, 2008
Posts: 7
|
(Msg. 1) Posted: Wed May 27, 2009 5:20 am
Post subject: VirtualHost bandwidth management
Archived from groups: linux>debian>isp (more info?)
|
|
|
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Hi All,
I'm looking for a way to measure data volume and bandwidth usage for
Apache virtualhosts. It would also be nice if we could throttle the
available bandwidth based on data volume usage.
I would prefer the data to be available in a database so our developers
can provide customers with a webpage showing how much data volume has
been used.
I tried many Apache modules for this purpose. Mod_cband used to work
fine, but it did not appear to work with MySQL databases. Since the
latest release of Debian, the Apache2 module mod_cband is no longer
available and the Etch version can be installed but does not seem to
work. Other modules do not provide the functionality I'm looking for.
Mod_bw makes it possible to limit bandwidth, but that's about it.
Another thing I tried is to parse awstats data and put it in a database.
This would work, but how should I throttle the bandwidth dynamically?
Also, we are planning to move all our vhosts to an LVS web cluster. A
few questions that arise are: what about statistics if many servers
serve the same pages? Which server should do the bandwidth measurement?
- --
Best regards, Met vriendelijke groet, Cordialement,
Steven Jacobs
UniWeb bvba
Your partner in professional web solutions.
UniWeb bvba, St-Amandsstraat 2, B-1853 Strombeek-Bever, Belgium, EU
tel: +32(0)2.3060000 | fax: +32(0)2.3060004 | e-mail: steven.TakeThisOut@uniweb.be
web: http://www.uniweb.eu
*Disclaimer:*
Current writing is under reservation of all rights and without any
prejudicial recognition. The contents of this e-mail and any attached
files are confidential and may be privileged or copyright by UniWeb. You
may only reproduce or distribute the material with our explicit
authorization. If you are not the intended recipient, please contact the
sender and immediately delete it and any copies of it from your system.
In this case copying, distributing or taking any action in reliance on
the contents of this information is strictly prohibited. Please note
that any opinions and views presented in this email are solely those of
the author and do not necessarily represent those of UniWeb bvba, unless
specifically stated.
General conditions: http://www.uniweb.eu/termsandconditions/
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iEYEARECAAYFAkoc/Y4ACgkQZ/DX0LN+5XVxsgCeO1g/JtHUXuXiUHiqHVib9eci
zYAAoKmbKNIxdT1vnB5qNjnxJ3p42spu
=aplH
-----END PGP SIGNATURE-----
--
To UNSUBSCRIBE, email to debian-isp-REQUEST.TakeThisOut@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster.TakeThisOut@lists.debian.org |
|
| Back to top |
|
 | |
External
Since: Dec 11, 2006
Posts: 65
|
(Msg. 2) Posted: Wed May 27, 2009 7:20 am
Post subject: Re: VirtualHost bandwidth management [Login to view extended thread Info.]
Archived from groups: per prev. post (more info?)
|
|
|
Steven Jacobs wrote:
> Hi All,
>
> I'm looking for a way to measure data volume and bandwidth usage for
> Apache virtualhosts. It would also be nice if we could throttle the
> available bandwidth based on data volume usage.
>
> I would prefer the data to be available in a database so our developers
> can provide customers with a webpage showing how much data volume has
> been used.
>
> I tried many Apache modules for this purpose. Mod_cband used to work
> fine, but it did not appear to work with MySQL databases. Since the
> latest release of Debian, the Apache2 module mod_cband is no longer
> available and the Etch version can be installed but does not seem to
> work. Other modules do not provide the functionality I'm looking for.
> Mod_bw makes it possible to limit bandwidth, but that's about it.
>
> Another thing I tried is to parse awstats data and put it in a database.
> This would work, but how should I throttle the bandwidth dynamically?
>
> Also, we are planning to move all our vhosts to an LVS web cluster. A
> few questions that arise are: what about statistics if many servers
> serve the same pages? Which server should do the bandwidth measurement?
Hi,
Just use libapache2-mod-log-sql-mysql that I maintain, it will:
- centralize all of your Apache logs in MySQL
- maintain a monthly statistic MySQL table
It is said to work with libdbi (that I also maintain), but I never
tried, as I don't use it.
All this is for bandwidth ACCOUNTING. Now if you need to do bandwidth
throttling, then there are many other apache modules for this. The issue
might be to connect the 2 together.
Please let me know your findings, and if we don't find anything, then I
think we might need to start to work on it (together?)! 
Thomas
--
To UNSUBSCRIBE, email to debian-isp-REQUEST RemoveThis @lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster RemoveThis @lists.debian.org |
|
| Back to top |
|
| |
External
Since: Nov 26, 2008
Posts: 7
|
(Msg. 3) Posted: Thu May 28, 2009 5:20 am
Post subject: Re: VirtualHost bandwidth management [Login to view extended thread Info.]
Archived from groups: per prev. post (more info?)
|
|
|
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Hi Thomas,
That is interesting, but I'm not the biggest fan of storing the logs in
a database. I like my logs clear text for ease of use. I know I could
write them both in the logs and in the db, but this seams overkill.
I do use mod_io and modified the logformat to show upstream and
downstream bandwidth usage for each request. The hard part is to
efficiently use this data for bandwidth control.
I can't get my head around how specialized shared hosting companies
enforce this. There has to be a way? Maybe there's a shaping appliance
or whatever, but I'd like to know what is is Preferably open source..
- --
Best regards, Met vriendelijke groet, Cordialement,
Steven Jacobs
UniWeb bvba
Thomas Goirand wrote:
> Steven Jacobs wrote:
>> Hi All,
>>
>> I'm looking for a way to measure data volume and bandwidth usage for
>> Apache virtualhosts. It would also be nice if we could throttle the
>> available bandwidth based on data volume usage.
>>
>> I would prefer the data to be available in a database so our developers
>> can provide customers with a webpage showing how much data volume has
>> been used.
>>
>> I tried many Apache modules for this purpose. Mod_cband used to work
>> fine, but it did not appear to work with MySQL databases. Since the
>> latest release of Debian, the Apache2 module mod_cband is no longer
>> available and the Etch version can be installed but does not seem to
>> work. Other modules do not provide the functionality I'm looking for.
>> Mod_bw makes it possible to limit bandwidth, but that's about it.
>>
>> Another thing I tried is to parse awstats data and put it in a database.
>> This would work, but how should I throttle the bandwidth dynamically?
>>
>> Also, we are planning to move all our vhosts to an LVS web cluster. A
>> few questions that arise are: what about statistics if many servers
>> serve the same pages? Which server should do the bandwidth measurement?
>
> Hi,
>
> Just use libapache2-mod-log-sql-mysql that I maintain, it will:
>
> - centralize all of your Apache logs in MySQL
> - maintain a monthly statistic MySQL table
>
> It is said to work with libdbi (that I also maintain), but I never
> tried, as I don't use it.
>
> All this is for bandwidth ACCOUNTING. Now if you need to do bandwidth
> throttling, then there are many other apache modules for this. The issue
> might be to connect the 2 together.
>
> Please let me know your findings, and if we don't find anything, then I
> think we might need to start to work on it (together?)!
>
> Thomas
>
>
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iEYEARECAAYFAkoeUqQACgkQZ/DX0LN+5XVn2QCeLk6MZPYdLnXtmIhoqXUUvKSP
5jsAoIzCiNUmcHrH8u6RYgnaNkqXqy9V
=ylN6
-----END PGP SIGNATURE-----
--
To UNSUBSCRIBE, email to debian-isp-REQUEST.TakeThisOut@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster.TakeThisOut@lists.debian.org |
|
| Back to top |
|
| |
External
Since: Nov 26, 2008
Posts: 7
|
(Msg. 4) Posted: Thu May 28, 2009 5:20 am
Post subject: Re: VirtualHost bandwidth management [Login to view extended thread Info.]
Archived from groups: per prev. post (more info?)
|
|
|
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Hi Brandon,
Sure, but we're moving to a solution where we can provide shared hosting
for a huge amount of users. We'll scale as we go. Up until now we've
only provided hosting for a handful of users, and your strategy worked.
Too bad this module is only available for cPanel though. We intend not
to use any off-the-shelf control panels for reasons of security and
manageability.
- --
Best regards, Met vriendelijke groet, Cordialement,
Steven Jacobs
UniWeb bvba
Brandon Helwig wrote:
> Most of the shared hosts use cPanel, which has a custom mod called
> mod_bwlimited
> Otherwise the ones that don't use cPanel probably have the resources to code
> their own. I've been looking for something like this as well, but have had
> no luck. So I guess my solution has been a policy of unlimited bandwidth,
> but don't abuse me or I'll shut you off. Not really been a problem so far.
>
> Brandon
>
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iEYEARECAAYFAkoeWcEACgkQZ/DX0LN+5XUfAACePEZolpV67HWQLM1Hfb5l47Mc
E64AoJuRgrM9SrNDyJtAMfBXSUCIBKWw
=RWod
-----END PGP SIGNATURE-----
--
To UNSUBSCRIBE, email to debian-isp-REQUEST.TakeThisOut@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster.TakeThisOut@lists.debian.org |
|
| Back to top |
|
| |
External
Since: May 28, 2009
Posts: 1
|
(Msg. 5) Posted: Thu May 28, 2009 7:20 am
Post subject: Re: VirtualHost bandwidth management [Login to view extended thread Info.]
Archived from groups: per prev. post (more info?)
|
|
|
hello,
Steven Jacobs wrote:
> Sure, but we're moving to a solution where we can provide shared hosting
> for a huge amount of users. We'll scale as we go. Up until now we've
> only provided hosting for a handful of users, and your strategy worked.
usually per-vhost bandwith management with huge amount of vhost require
very very very huge investments on hardware (and infrasctructures, of
course), that's why many hosting company prefer other way to limits users;
1) no limits on the tech side, but if you user is using so much resource
that it's creating problmes you may block him (as it's stated in the
AUP he accepted)
2) bottlenek limitations, eg put all the customers with the same sla on
the same server (or group of servers) and let them to fight for the
resources (of course more the money the customer give you less the
number of concurrent users you put on a single server)
3) commercially limit the amount ot traffic (how many gigabyte per
month) and not the bandwidth (megabit per seconds); of course you have
to disable the user (or bill him the extra) when the user exceed his
allocated traffic.
if you have huge amount of users you also have to use a big number of
servers with some load balancer in front of them, so if you have to use
bandwidth limit per-vhost you have to put this limits on the load
balancer (or on a further inline device/server) not on the apache, but
as for as I know, there is no opensource reverse-proxy able to manage
huge amount of traffic with L7 bandwith management enabled
--
bye,
emilio
--
To UNSUBSCRIBE, email to debian-isp-REQUEST.RemoveThis@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster.RemoveThis@lists.debian.org |
|
| Back to top |
|
| |
External
Since: Jun 06, 2009
Posts: 1
|
(Msg. 6) Posted: Sat Jun 06, 2009 3:20 am
Post subject: Posting style on Debian mailing lists (was Re: VirtualHost bandwidth management) [Login to view extended thread Info.]
Archived from groups: per prev. post (more info?)
|
|
|
[Please don't top-post. Instead, remove quoted material that you're
not replying to, and reply inline to the points you're responding to.]
On Fri, May 29, 2009 at 02:18:10AM +0200, Daniel Hood wrote:
> I think most of the hosts that don't use CPanel and some how limit it, use
> QoS on a router infront of the servers. Im sure you can do this with a
> Debian box somehow. I've done with a Cisco router, but that is freaking
> expensive. Im sure you can limit bandwidth and monitor bandwidth via QoS
> (Quality of Service) solutions. I know debian has many ways to implement
> QoS. Its just finding the right way.
snipped 37 lines!!
Hey guys,
Trimming [1], ie. cutting of unneeded content in replies is also
generally prefered.
As is inline replying [2].
Use common sense to make it as simple as possible to others to read your
post. Most people on this list have the thread present (either in their
mail program or via the web) and therefore it is _not_ wanted to resend
the whole conversation over and over again (as is common practise in
some commercial environments).
[1] http://en.wikipedia.org/wiki/Posting_style#Trimming_and_reformatting
[2] http://en.wikipedia.org/wiki/Posting_style#Inline_replying
--
Chris.
======
I contend that we are both atheists. I just believe in one fewer god
than you do. When you understand why you dismiss all the other
possible gods, you will understand why I dismiss yours.
-- Stephen F Roberts
--
To UNSUBSCRIBE, email to debian-isp-REQUEST.DeleteThis@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster.DeleteThis@lists.debian.org |
|
| Back to top |
|
| |
External
Since: Nov 26, 2008
Posts: 7
|
(Msg. 7) Posted: Mon Jun 15, 2009 9:20 am
Post subject: Re: VirtualHost bandwidth management [Login to view extended thread Info.]
Archived from groups: per prev. post (more info?)
|
|
|
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Hi Maarten,
I liked mod_cband, but further down in my post I explained the reasons
why this module does not work anymore under Debian Lenny, nor the latest
versions of Ubuntu.
I think I'm just gonna use mod_logio, parse the logs for each vhost and
add this data to a database. This at least takes care of the accounting
part. Users that ignore the fair use policy will get banned.
- --
Best regards, Met vriendelijke groet, Cordialement,
Steven Jacobs
UniWeb bvba
Maarten Vink wrote:
>
> On 27 mei 2009, at 10:45, Steven Jacobs wrote:
>
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA1
>>
>> Hi All,
>>
>> I'm looking for a way to measure data volume and bandwidth usage for
>> Apache virtualhosts. It would also be nice if we could throttle the
>> available bandwidth based on data volume usage.
>
> Have a look at mod_cband. It's packaged for Debian, and allows you to
> limit both transfer speeds and/or limits for the total traffic volume of
> a website. There's a webbased status page that your support staff can
> use when diagnosing problems as well, so no need to write something
> yourself.
>
> http://www.howtoforge.com/mod_cband_apache2_bandwidth_quota_throttling
> has some instructions on setting things up.
>
> Regards,
>
> Maarten Vink
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iEYEARECAAYFAko2SSYACgkQZ/DX0LN+5XXRMgCdFzKhZRmpQ/UJqRS4SkAAM3El
YW8AoK7rdpWl7AyVirtTXjMlk/UKoUqK
=RkOP
-----END PGP SIGNATURE-----
--
To UNSUBSCRIBE, email to debian-isp-REQUEST.RemoveThis@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster.RemoveThis@lists.debian.org |
|
| Back to top |
|
| |
External
Since: Dec 11, 2006
Posts: 65
|
(Msg. 8) Posted: Mon Jun 15, 2009 3:20 pm
Post subject: Re: VirtualHost bandwidth management [Login to view extended thread Info.]
Archived from groups: per prev. post (more info?)
|
|
|
Steven Jacobs wrote:
> Hi Maarten,
>
> I liked mod_cband, but further down in my post I explained the reasons
> why this module does not work anymore under Debian Lenny, nor the latest
> versions of Ubuntu.
How did you try? Did you actually try the binary module from Etch, or
did you try to recompile it? Did you go to the BTS to see why exactly
mod_cband has been removed?
This seems to be a very interesting piece of software. If it's gone from
Debian just because the old maintainer didn't had time or wish to
continue his work on it, then I'm willing to take over and maintain it.
But if it's because the module has (new) issues, then I don't know what
I can do. Can you try to find out what's going on? Maybe a quick email
to the old maintainer will help...
> I think I'm just gonna use mod_logio, parse the logs for each vhost and
> add this data to a database. This at least takes care of the accounting
> part. Users that ignore the fair use policy will get banned.
Then if you want to do this, you'd better off using mod_log_sql that I
maintain. It does the above but in real time, maintaining a monthly
score board for all domain names.
Now, seeing both mod_log_sql and mod_cband, I am seeing that it might be
possible to combine both 2 and make a unique one, that would be able to
both log into sql in a score board (so it's convenient for displaying
the score board to the user), and do bandwidth limiting.
All this is VERY interesting, and I think that gives us very good tracks
for improving our control panel. Thanks for raising this topic and
insisting in giving me the pointer of mod_cband!!!
Programming apache modules is a bit tricky, form my experience (I don't
really like doing it), and it needs a lot of testing to make sure that
there's no memory leaks, and stuffs like that. But the result can be
really cool. Steven, would you like to work with me together in the
development of a mod_cband + mod_log_sql in a single module? That would
be the dream tool for shared hosting bandwidth management! Are you able
to do C developments, or at least help?
Thomas
--
To UNSUBSCRIBE, email to debian-isp-REQUEST RemoveThis @lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster RemoveThis @lists.debian.org |
|
| Back to top |
|
| |
External
Since: Dec 11, 2006
Posts: 65
|
(Msg. 9) Posted: Thu Jun 18, 2009 1:20 pm
Post subject: Re: VirtualHost bandwidth management [Login to view extended thread Info.]
Archived from groups: per prev. post (more info?)
|
|
|
Steven Jacobs wrote:
> Hi Thomas,
>
> Thanks for your hard work
> We are currently using only Etch and Ubuntu Hardy servers for the moment.
>
> I'll try to find the time to make a package using the latest version and
> test it.
???
I just wrote that this is exactly what I did...
ftp://ftp.gplhost.com/debian/pool/lenny/main/m/mod-cband/
Thomas
--
To UNSUBSCRIBE, email to debian-isp-REQUEST RemoveThis @lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster RemoveThis @lists.debian.org |
|
| Back to top |
|
| |
External
Since: Nov 26, 2008
Posts: 7
|
(Msg. 10) Posted: Fri Jun 19, 2009 5:20 am
Post subject: Re: VirtualHost bandwidth management [Login to view extended thread Info.]
Archived from groups: per prev. post (more info?)
|
|
|
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Hi Thomas,
The package does not work for Etch, nor does it for Ubuntu Hardy.
Currently I have a lot on my hands, but I'll definately test it next week!
Thomas Goirand wrote:
> Steven Jacobs wrote:
>> Hi Thomas,
>>
>> Thanks for your hard work
>> We are currently using only Etch and Ubuntu Hardy servers for the moment.
>>
>> I'll try to find the time to make a package using the latest version and
>> test it.
>
> ???
>
> I just wrote that this is exactly what I did...
>
> ftp://ftp.gplhost.com/debian/pool/lenny/main/m/mod-cband/
>
> Thomas
>
>
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iEYEARECAAYFAko7VJAACgkQZ/DX0LN+5XWGxwCaA3rSL/LW9e+iskruwC0PFAvP
CBUAnRSICx39jl3PGkIEW7w80buGwTdI
=x7Zz
-----END PGP SIGNATURE-----
--
To UNSUBSCRIBE, email to debian-isp-REQUEST DeleteThis @lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster DeleteThis @lists.debian.org |
|
| Back to top |
|
| |
External
Since: Dec 11, 2006
Posts: 65
|
(Msg. 11) Posted: Fri Jun 19, 2009 9:20 am
Post subject: Re: VirtualHost bandwidth management [Login to view extended thread Info.]
Archived from groups: per prev. post (more info?)
|
|
|
Steven Jacobs wrote:
> Hi Thomas,
>
> The package does not work for Etch, nor does it for Ubuntu Hardy.
> Currently I have a lot on my hands, but I'll definately test it next week!
Of course, it's compiled for Debian Lenny!
If you need to recompile it, here's what to do.
1- Add in your sources.list:
deb-src ftp://ftp.gplhost.com/debian lenny main
2- apt-get install build-essential debhelper autotools-dev
apache2-prefork-dev po-debconf apache2.2-common apache2 apache2-prefork
3- apt-get source libapache2-mod-cband
4- cd mod-cband-0.9.7.5
5- dpkg-buildpackage
6- cd ..
Install the resulting package.
Thomas
--
To UNSUBSCRIBE, email to debian-isp-REQUEST.TakeThisOut@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster.TakeThisOut@lists.debian.org |
|
| Back to top |
|
| |
| Related Topics: | [gentoo-user] Gentoo apache2 VirtualHost :443 config - I need to include a Location block for one subdomain, but not for another. I have separate VirtualHost blocks set up..
Looking for bandwidth "counter" - I'm looking for something that will give me bandwidth totals for each day/week/month, etc. webminstats has a really nic...
Bandwidth limit - Hi! I am running RH8 as a router for LAN. I would like that my LAN users get equal amount of bandwidth because now..
Bandwidth monitor - Hi there Just wondering what would be a good free live bandwidth/net trafic monitor for suse linux, Eg: DUmeter for....
bandwidth analyzer for linux - I'm have a linux box sitting on a hub in promiscious mode between my cable modem and my Linksys router. i have Snort..
Bandwidth Monitor in Squid - Hi, Anyone plz tell me how to identify the bandwidth usage monitor under squid proxy,, Regards Sivakumar |
|
You can post new topics in this forum
You can reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
|
|
|
|
|
FAQ
Search
Profile
Private Messages
Log in
Linux