Password file with over 3000 users.

Hi
Ian schrieb:
> ...
> - Must be pam compatible.
>
> - Most users have MD5 passwords, but some are still crypt passwords. I
> do not have ready access to the original plain text passwords.
>
> - I don't want to mess around with too much, so I must have real "Unix
> accounts" (UID's and home directories) for each user.
>
> - Vanilla Debian "deb" packages.
>
> What alternatives should I consider?
Ever thougt about using LDAP? I do not have an installation that is as
big as yours (only about 800 users), but I have no problems with
performance and the server is not the fastest. home directorys are even
exported with nfs and samba PDC with LDAP and Samba for a M$ Domain) is
doing its job too on this box.


best regards


Bernd

_____________________________________________________________________


--
To UNSUBSCRIBE, email to debian-isp-REQUEST.TakeThisOut@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster.TakeThisOut@lists.debian.org

Hello,

Marc Schiffbauer a écrit (Tue, Sep 18, 2007 at 06:24:34PM +0200) :
> Installing nscd might be worth a try.
> I *can* speedup your system.

nscd *sigh* ... As we use LDAP, we need this beast that crashes
happily several times a day.
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=344563

We need a every-5' cronjob to restart it if it has disappeared.

rm -f /var/db/nscd/*; /etc/init.d/nscd start

In other words: I can't really recommend nscd.
YMMV.

--
Emmanuel Halbwachs
Resp. Réseau/Sécurité Observatoire de Paris-Meudon
tel : (+33)1 45 07 75 54 5 Place Jules Janssen
fax : (+33)1 45 07 76 13 F 92195 MEUDON CEDEX


--
To UNSUBSCRIBE, email to debian-isp-REQUEST RemoveThis @lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster RemoveThis @lists.debian.org

Emmanuel Halbwachs wrote:
> Hello,
>
> Marc Schiffbauer a écrit (Tue, Sep 18, 2007 at 06:24:34PM +0200) :
>
>> Installing nscd might be worth a try.
>> I *can* speedup your system.
>>
>
> nscd *sigh* ... As we use LDAP, we need this beast that crashes
> happily several times a day.
> http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=344563
>
> We need a every-5' cronjob to restart it if it has disappeared.
>
> rm -f /var/db/nscd/*; /etc/init.d/nscd start
>
> In other words: I can't really recommend nscd.
> YMMV.
>
>
I've managed several OpenLDAP setups, with nscd and several thousand
users, the key is to properly configure nscd, which needs a properly
sized hash table to operate.

/etc/nscd.conf has a suggested-size parameter for passwd, it should be a
prime number, we chose a number bigger than twice the size of our
expected user base. Its default is 211, which is too small for lots of
users.

This way, nscd usually stops failing (usually, it can fail if you set
it up for more databases than needed, like hosts). You can use the
bsdgames "primes" program to get the hash size.

--
Hector Gonzalez

--On September 18, 2007 6:41:10 PM +0200 Emmanuel Halbwachs
<Emmanuel.Halbwachs DeleteThis @obspm.fr> wrote:

> Hello,
>
> Marc Schiffbauer a écrit (Tue, Sep 18, 2007 at 06:24:34PM +0200) :
>> Installing nscd might be worth a try.
>> I *can* speedup your system.
>
> nscd *sigh* ... As we use LDAP, we need this beast that crashes
> happily several times a day.
> http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=344563

We're experieincing occasional nscd crashes in our newest etch based
systems, but never in the older ones. and the fact that nscd is doing any
sort of persistence bugs me too, it shouldn't. when it crashes on our
webservers the slapd then starts to work really really hard keeping up with
all the requests.

>
> We need a every-5' cronjob to restart it if it has disappeared.
>
> rm -f /var/db/nscd/*; /etc/init.d/nscd start
>
> In other words: I can't really recommend nscd.
> YMMV.
>
> --
> Emmanuel Halbwachs
> Resp. Réseau/Sécurité Observatoire de Paris-Meudon
> tel : (+33)1 45 07 75 54 5 Place Jules Janssen
> fax : (+33)1 45 07 76 13 F 92195 MEUDON CEDEX
>
>
> --
> To UNSUBSCRIBE, email to debian-isp-REQUEST DeleteThis @lists.debian.org
> with a subject of "unsubscribe". Trouble? Contact
> listmaster DeleteThis @lists.debian.org
>
>
>



--
Michael Loftis
Modwest Operations Manager
Powerful, Affordable Web Hosting

On Tue, Sep 18, 2007 at 05:19:15PM +0200, Ian wrote:
> I have a server which runs with normal passwd/shadow/group/gshadow
> files. There are now over 3000 users and I am beginning to notice a
> performance slowdown and it is about time I looked for something more
> efficient - I like the idea of db or cdb database files.
>
> My requirements:
>
> - Must be pam compatible.
>
> - Most users have MD5 passwords, but some are still crypt passwords. I
> do not have ready access to the original plain text passwords.
>
> - I don't want to mess around with too much, so I must have real "Unix
> accounts" (UID's and home directories) for each user.
>
> - Vanilla Debian "deb" packages.
>
> What alternatives should I consider?

if you have the libnss-db package (part of nsswitch) installed, you have
everything you need already.

nsswitch.conf allows you to specify "db" as one of the sources for
passwd, shadow, group, and other files. they are read from a Berkeley
db file in /var/lib/misc (i.e. a quick indexed lookup rather than a
sequential search).

there is also a Makefile in /var/lib/misc for generating the .db versions.

to set up:

1. edit /etc/nsswitch.conf and insert the word "db" into the lines that you
want to use the db module.

e.g. change this:

passwd: compat
group: compat
shadow: compat

to this:

passwd: db compat
group: db compat
shadow: db compat


2. edit /etc/default/libnss-db to tell it which dbs to generate.

3. run "cd /var/lib/misc ; make"

4. set up a cron job (to run, say, every 5 minutes) to run the commands
in step 3 (i.e. "cd /var/lib/misc ; make"). you probably want to
redirect stdout and stderr to /dev/null so that root's mail doesn't get
flooded with make's output (mostly "make: Nothing to be done for `all'."
unless passwd,group, or shadow have changed)


see /usr/share/doc/libnss-db and other related documentation for more info.



NOTE: LDAP is also a good alternative, but a *LOT* more work to set up.
libnss-db is a simple way to speed up what already works by putting the
passwd etc files into hashed database files.


craig

PS: this works. i did this several years ago on one server when the number of
accounts grew to about 5000. there is one small catch - with the cron job
running every 5 minutes, there is a small window of time when the source files
in /etc have been updated but the .db versions haven't been regenerated yet.

the nsswitch.conf file will check both the db and the original source files in
order, so it does not prevent new accounts from logging in. for account
deletions, however, the deleted account will still work until the .db files
are regenerated. similarly, password changes will not take effect
immediately.

actually, it's been years - i can't remember if only the old password
(in /var/lib/misc/shadow.db) works, or if both the old (shadow.db) and
new (/etc/shadow) password will work. either way, that's only until the
cron job runs make again (i.e. at most, up to 5 minutes. or less if you
have cron run make more frequently).

if you have written scripts to assist with account
creation/deletion/changing, you could easily modify them to run "cd
/var/lib/misc ; make" after any change, thus eliminating the delay.

you still want the cron job, though, in case there are other ways for a
password to be changed - shell login by users or poppassd or samba, for
instance.



--
craig sanders <cas RemoveThis @taz.net.au>


--
To UNSUBSCRIBE, email to debian-isp-REQUEST RemoveThis @lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster RemoveThis @lists.debian.org

--On September 18, 2007 5:19:15 PM +0200 Ian <iforbes.RemoveThis@zsd.co.za> wrote:

> Hi All
>
> I have a server which runs with normal passwd/shadow/group/gshadow
> files. There are now over 3000 users and I am beginning to notice a
> performance slowdown and it is about time I looked for something more
> efficient - I like the idea of db or cdb database files.

You've got two issues to solve here. Authentication you actually have more
options than UID/Groups mapping which is controlled by the name service
swith library, libnss. LDAP can solve both using libnss-ldap, modifying
/etc/ldap.conf and /etc/nsswitch.conf and then importing your passwd file
into LDAP. That's actually what we use here, though we modified the
'stock' LDAP configuration to allow a substr index on one of the fields
that normally isn't allowed by the LDAP schema. It was causing the LDAP
servers to thrash. The solution is seamless to the users, the LDAP system
supports crypt or md5 based passwords as normal since the password calls,e
tc, are all handled via the normal pam_unix library via libnss, which is
exactly how it does it anyway. The only difference is where it's obtaining
the users.

There also might be a way (and i haven't looke dinto this) to get Linux to
use a GDB/BDB passwd.db like *BSD's do.



>
> My requirements:
>
> - Must be pam compatible.
>
> - Most users have MD5 passwords, but some are still crypt passwords. I
> do not have ready access to the original plain text passwords.
>
> - I don't want to mess around with too much, so I must have real "Unix
> accounts" (UID's and home directories) for each user.
>
> - Vanilla Debian "deb" packages.
>
> What alternatives should I consider?
>
> Thanks
>
>
> Ian
>
>
> --
> To UNSUBSCRIBE, email to debian-isp-REQUEST.RemoveThis@lists.debian.org
> with a subject of "unsubscribe". Trouble? Contact
> listmaster.RemoveThis@lists.debian.org
>
>
>



--
Michael Loftis
Modwest Operations Manager
Powerful, Affordable Web Hosting


--
To UNSUBSCRIBE, email to debian-isp-REQUEST.RemoveThis@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster.RemoveThis@lists.debian.org

--On September 19, 2007 9:35:50 AM +1000 Craig Sanders <cas DeleteThis @taz.net.au>
wrote:


>
> NOTE: LDAP is also a good alternative, but a *LOT* more work to set up.
> libnss-db is a simple way to speed up what already works by putting the
> passwd etc files into hashed database files.

I will totally agree there and for a single system LDAP can be severe
overkill. libnss-db is exactly what i was mentioning in my just previous
email.

>
>
> craig
>
> PS: this works. i did this several years ago on one server when the
> number of accounts grew to about 5000. there is one small catch - with
> the cron job running every 5 minutes, there is a small window of time
> when the source files in /etc have been updated but the .db versions
> haven't been regenerated yet.
>
> the nsswitch.conf file will check both the db and the original source
> files in order, so it does not prevent new accounts from logging in. for
> account deletions, however, the deleted account will still work until the
> .db files are regenerated. similarly, password changes will not take
> effect immediately.
>
> actually, it's been years - i can't remember if only the old password
> (in /var/lib/misc/shadow.db) works, or if both the old (shadow.db) and
> new (/etc/shadow) password will work. either way, that's only until the
> cron job runs make again (i.e. at most, up to 5 minutes. or less if you
> have cron run make more frequently).
>
> if you have written scripts to assist with account
> creation/deletion/changing, you could easily modify them to run "cd
> /var/lib/misc ; make" after any change, thus eliminating the delay.
>
> you still want the cron job, though, in case there are other ways for a
> password to be changed - shell login by users or poppassd or samba, for
> instance.
>
>
>
> --
> craig sanders <cas DeleteThis @taz.net.au>
>
>
> --
> To UNSUBSCRIBE, email to debian-isp-REQUEST DeleteThis @lists.debian.org
> with a subject of "unsubscribe". Trouble? Contact
> listmaster DeleteThis @lists.debian.org
>
>
>



--
Michael Loftis
Modwest Operations Manager
Powerful, Affordable Web Hosting


--
To UNSUBSCRIBE, email to debian-isp-REQUEST DeleteThis @lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster DeleteThis @lists.debian.org

Craig Sanders wrote:
> 4. set up a cron job (to run, say, every 5 minutes) to run the
> commands
> in step 3 (i.e. "cd /var/lib/misc ; make"). you probably want to
> redirect stdout and stderr to /dev/null so that root's mail doesn't
> get flooded with make's output (mostly "make: Nothing to be done for
> `all'." unless passwd,group, or shadow have changed)

This is what I added to my /etc/crontab file:

*/5 * * * * root cd /var/lib/misc && make | grep -v "make: Nothing to
be done for .all."

If changes do occur, then I will see output from them, although the output
doesn't seem that meaningful -- as in you don't see which user changed
his/her password....

Kind Regards
AndrewM

Andrew McGlashan
Broadband Solutions now including VoIP

Current Land Line No: 03 9912 0504
Mobile: 04 2574 1827 Fax: 03 8790 1224

National No: 1300 85 3804

Affinity Vision Australia Pty Ltd
http://www.affinityvision.com.au
http://adsl2choice.net

In Case of Emergency -- http://www.affinityvision.com.au/ice.html


--
To UNSUBSCRIBE, email to debian-isp-REQUEST.DeleteThis@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster.DeleteThis@lists.debian.org

Craig Sanders wrote:

> if you have the libnss-db package (part of nsswitch) installed, you have
> everything you need already.

Thanks. This is exactly what I need. I actually found this and got it
running shortly after I posted my original message. Everything is in
Debian - except the documentation!

My setup is pretty much exactly as you recommend.

> PS: this works. i did this several years ago on one server when the number of
> accounts grew to about 5000. there is one small catch - with the cron job
> running every 5 minutes, there is a small window of time when the source files
> in /etc have been updated but the .db versions haven't been regenerated yet.

Just one small addition.

I added this line to end of our /usr/local/sbin/adduser.local

[ -e /var/lib/misc/Makefile ] && cd /var/lib/misc/ && make

And I have a cron job which runs the same command once per hour to
effect deletions and account locking etc.

I will have to look into controlling the cron spam.

We actually use this password file across a couple of servers with a
cron job that copies it across with rsync every 5 minutes. At some stage
I must look at a SQL or LDAP based solution. I originally chose the
rsync because each server can run on its own if one goes down and there
are no performance issues. But libpam-ccreds and nss-updatedb appear to
offer the same functionality when coupled with ldap.

We already have a postgresql backend for our radius server, would it be
better to run SQL -> LDAP -> nss or go directly from SQL -> nss?

Thanks

Ian






--
To UNSUBSCRIBE, email to debian-isp-REQUEST.DeleteThis@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster.DeleteThis@lists.debian.org

On Fri, Sep 21, 2007 at 12:02:23AM +0200, Ian wrote:
> Craig Sanders wrote:
>
> > if you have the libnss-db package (part of nsswitch) installed, you have
> > everything you need already.
>
> Thanks. This is exactly what I need. I actually found this and got it
> running shortly after I posted my original message. Everything is in
> Debian - except the documentation!

AFAIK, there isn't any documentation. certainly not any step-by-step howto.

it's one of the many things that IF you know about it, it's easy to
figure out. the difficult part is finding out about it in the first
place.


> I will have to look into controlling the cron spam.

append ">/dev/null 2>&1" to the cron job. IMO, the output of that
Makefile isn't ever interesting enough to care about (and if there's a
problem, you'll notice it long before you read root's mail).


> We actually use this password file across a couple of servers with a
> cron job that copies it across with rsync every 5 minutes. At some
> stage I must look at a SQL or LDAP based solution. I originally chose
> the rsync because each server can run on its own if one goes down and
> there are no performance issues. But libpam-ccreds and nss-updatedb
> appear to offer the same functionality when coupled with ldap.
>
> We already have a postgresql backend for our radius server, would it
> be better to run SQL -> LDAP -> nss or go directly from SQL -> nss?

can't say. personally, i tend to avoid making web/mail/shell/etc servers
dependant on an SQL server for authentication (especially if it's
mysql). i'd be more inclined to use LDAP and have an LDAP slave on
every server that needs it.

SQL servers are useful, but i just don't trust them the way i trust
plain text files.

even where i've stored, e.g., postfix maps in postgresql, i've always
set them up so that the postgresql table is dumped (by cron job) to
a plain text file and made into a hashed db file - that way the mail
server will keep on working even if the postgresql server is down or
unreachable or overloaded.

the trick to that is using a timestamp table auto-updated by a trigger
when anything in the database is changed - that way you can avoid
regenerating the hash maps when nothing has changed (i.e. the same idea
as timestamp dependancies in Make).


craig

--
craig sanders <cas.TakeThisOut@taz.net.au>


--
To UNSUBSCRIBE, email to debian-isp-REQUEST.TakeThisOut@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster.TakeThisOut@lists.debian.org

--On September 21, 2007 12:02:23 AM +0200 Ian <iforbes RemoveThis @zsd.co.za> wrote:

> Craig Sanders wrote:
>
>> if you have the libnss-db package (part of nsswitch) installed, you have
>> everything you need already.
>
> Thanks. This is exactly what I need. I actually found this and got it
> running shortly after I posted my original message. Everything is in
> Debian - except the documentation!
>
> My setup is pretty much exactly as you recommend.
>
>> PS: this works. i did this several years ago on one server when the
>> number of accounts grew to about 5000. there is one small catch - with
>> the cron job running every 5 minutes, there is a small window of time
>> when the source files in /etc have been updated but the .db versions
>> haven't been regenerated yet.
>
> Just one small addition.
>
> I added this line to end of our /usr/local/sbin/adduser.local
>
> [ -e /var/lib/misc/Makefile ] && cd /var/lib/misc/ && make


>
> And I have a cron job which runs the same command once per hour to
> effect deletions and account locking etc.
>
> I will have to look into controlling the cron spam.

2>&1 > /dev/null to end of command line.

>
> We actually use this password file across a couple of servers with a
> cron job that copies it across with rsync every 5 minutes. At some stage
> I must look at a SQL or LDAP based solution. I originally chose the
> rsync because each server can run on its own if one goes down and there
> are no performance issues. But libpam-ccreds and nss-updatedb appear to
> offer the same functionality when coupled with ldap.
>
> We already have a postgresql backend for our radius server, would it be
> better to run SQL -> LDAP -> nss or go directly from SQL -> nss?

Up to you to me it seems simplest to run LDAP via it's builtin
bdb/ldbm/gdbm facilities. You can run replica slaves on the other
machines. It's what we do here. We're working on improving security by
doing authentication in a different method so we're not syncing auth
information elsewhere but have account data available everywhere. (we've
something like 10 webservers that each have LDAP slaves, another like 5 or
6 mail servers, etc).

You also have to be careful not to get service dependencies (like you need
to make sure your mysql and ldap server daemons users aren't in LDAP) when
you start making services depend on eachother.


--
To UNSUBSCRIBE, email to debian-isp-REQUEST RemoveThis @lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster RemoveThis @lists.debian.org

Quoting Michael Loftis <mloftis.DeleteThis@modwest.com>:

>> I will have to look into controlling the cron spam.
>
> 2>&1 > /dev/null to end of command line.

Actually, it's the other way around:

> /dev/null 2>&1

You first have to redirect STDOUT, then tell STDERR to go to STDOUT...


--
To UNSUBSCRIBE, email to debian-isp-REQUEST.DeleteThis@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster.DeleteThis@lists.debian.org

Turbo Fredriksson wrote:
> Quoting Michael Loftis <mloftis.DeleteThis@modwest.com>:
>
>>> I will have to look into controlling the cron spam.
>> 2>&1 > /dev/null to end of command line.
> Actually, it's the other way around:
>
>> /dev/null 2>&1
>
> You first have to redirect STDOUT, then tell STDERR to go to STDOUT...

I've heard that before many times, but the reverse has never failed me.

Kind Regards
AndrewM

Andrew McGlashan
Broadband Solutions now including VoIP

Current Land Line No: 03 9912 0504
Mobile: 04 2574 1827 Fax: 03 8790 1224

National No: 1300 85 3804

Affinity Vision Australia Pty Ltd
http://www.affinityvision.com.au
http://adsl2choice.net

In Case of Emergency -- http://www.affinityvision.com.au/ice.html


--
To UNSUBSCRIBE, email to debian-isp-REQUEST.DeleteThis@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster.DeleteThis@lists.debian.org