s4:provision Rework and further automate setup of OpenLDAP backend

heres the summary of all changes/extensions:

- Andrew Bartlett's patch to generate indext
- Howard Chu's idea to use nosync on the DB included, but made optional

- slaptest-path is not needed any more (slapd -Ttest is used instead)
and is therefore removed. slapd-path is now recommended when
openldap-backend is chosen.
its also used for olc-conversion

- slapd-detection is now always done by ldapsearch (ldb module),
looking anonymous for objectClass: OpenLDAProotDSE via our ldapi_uri.

- if ldapsearch was not successfull, (no slapd listening on our socket)
slapd is
started via special generated slapdcommand_prov  (ldapi_uri only)

- slapd-"provision-process" startup is done via pythons subprocess.

- the slapd-provision-pid is stored under paths.ldapdir/slapd_provision_pid.

- after provision-backend is finished:
--- slapd.pid is compared with our stored slapd_provision_pid.
if the are unique, slapd.pid will be read out, and the
slapd "provison"-process will be shut down.
--- proper slapd-shutdown is verified again with ldb-search -> ldapi_uri
-> rootDSE.
--- if the pids are different or one of the pid-files is missing, slapd
will not be shut down,
instead an error message is displayed to locate slapd manually
--- extended help-messages (relevant to slapd) are always displayed,
e.g. the commandline with which slapd has to be started when everythings
finished
(slapd-commandline is stored under paths.ldapdir/slapd_command_file.txt))

- upgraded the content of the mini-howto (howto-ol-backend-s4.txt)

howto-ol-backend-s4.txt

@@ -0,0 +1,177 @@
+Samba4  OpenLDAP-Backend Quick-Howto
+====================================
+
+[email protected]  -  August 2009
+
+
+This Mini-Howto describes in a very simplified way 
+how to setup Samba 4 (S4) (pre)Alpha 9 with the
+OpenLDAP (OL) -Backend.
+Use of OpenLDAP >= 2.4.17 is strongly recommended.
+
+
+1.) Download and compile OpenLDAP. 
+
+The use of (older) Versions shipped with Distributions often
+causes trouble, so dont use them. Configure-Example:
+
+#> ./configure --enable-overlays=yes --with-tls=yes --with-cyrus-sasl=yes
+#> make depend && make && make install
+
+Note: openssl and cyrus-sasl libs should be installed
+before compilation.
+
+
+
+2.) Prepare S4 to use OL-Backend:
+
+Run the provision-backend Python-Script first, then "final" provision
+(these 2-step process will be merged in the future)
+
+Simple provision-backend Example:
+
+#> setup/provision-backend --realm=ldap.local.site \
+  --domain=LDAP --ldap-admin-pass="linux" \
+  --ldap-backend-type=openldap \
+  --server-role='domain controller' \
+  --ol-slapd="/usr/local/libexec"
+
+After that, you should get a similar output:
+
+--------
+Your openldap Backend for Samba4 is now configured, and is ready to be started
+Server Role:         domain controller
+Hostname:            ldapmaster
+DNS Domain:          ldap.local.site
+Base DN:             DC=ldap,DC=local,DC=site
+LDAP admin user:     samba-admin
+LDAP admin password: linux
+LDAP Debug-Output:
+(1, 'connection to remote LDAP server dropped?')
+Ok. - No other slapd-Instance listening on: ldapi://%2Fusr%2Flocal%2Fsamba%2Fprivate%2Fldap%2Fldapi. Starting al provision.
+Started slapd for final provisioning with PID: 21728
+
+Now run final provision with: --ldap-backend=ldapi --ldap-backend-type=openldap --password=linux --username=sa=ldap.local.site --domain=LDAP --server-role='domain controller'
+
+--------
+
+Since this (pre)Alpha, you dont have to run slapd manually
+any more. slapd will be started automatically, when 
+provision-backend is done, listening on the
+ldapi://-Socket. System should be ready 
+for final provision now:
+
+
+3.) Final provision:
+
+Use the Parameters displayed above to run final provision.
+(you can add --adminpass=<yourpass> to the parameters,
+otherwise a random password will be generated for 
+cn=Administrator,cn=users,<Your Base-DN>):
+
+#> setup/provision --ldap-backend=ldapi \
+   --ldap-backend-type=openldap --password=linux \
+   --username=samba-admin --realm=ldap.local.site \
+   --domain=LDAP --server-role='domain controller'\
+   --adminpass=linux
+
+At the End of the final provision you should get
+the following output (only partial here). Read it carefully:
+
+--------
+...
+A Kerberos configuration suitable for Samba 4 has been generated at /usr/local/samba/private/krb5.conf
+LDAP Debug-Output:[Message({'dn': Dn(''), 'objectClass': MessageElement(['top','OpenLDAProotDSE'])})]
+slapd-PID-File found. PID is :21728
+
+File from provision-backend with stored PID found. PID is :21728
+
+slapd-Process used for provisioning with PID: 21728
+ will now be shut down.
+slapd-Process used for final provision was properly shut down.
+Use later the following commandline to start slapd, then Samba:
+/usr/local/libexec/slapd -f /usr/local/samba/private/ldap/slapd.conf -h ldapi://%2Fusr%2Flocal%2Fsamba%2Fprivate%2Fldap%2Fldapi
+
+This slapd-Commandline is also stored under: /usr/local/samba/private/ldap/slapd_command_file.txt
+Please install the phpLDAPadmin configuration located at /usr/local/samba/private/phpldapadmin-config.php into /etc/phpldapadmin/config.php
+Once the above files are installed, your Samba4 server will be ready to use
+Server Role:    domain controller
+Hostname:       ldapmaster
+NetBIOS Domain: LDAP
+DNS Domain:     ldap.local.site
+DOMAIN SID:     S-1-5-21-429312062-2328781357-2130201529
+Admin password: linux
+
+--------
+
+Our slapd in "provision-mode" wiil be shut down automatically 
+after final provision ends.
+
+
+4.) Run OL and S4:
+
+After you completed the other necessary steps (krb and named-specific),
+start first OL with the commandline displayed in the output under (3),
+(remember: the slapd-Commandline is also stored in the file ../slapd_command_file.txt)
+then S4.
+
+
+
+5.) Special Setup-Types:
+
+a) OpenLDAP-Online Configuration (olc):
+Use the provision-backend Parameter 
+
+ --ol-olc=yes.
+
+In that case, the olc will be setup automatically
+under ../private/slapd.d/.
+olc is accessible via "cn=samba-admin,cn=samba" and Base-DN "cn=config"
+olc is intended primarily for use in conjunction with MMR
+
+Attention: You have to start OL with the commandline
+displayed in the output under (3), but you have to set a 
+listening port of slapd manually:
+
+(e.g. -h ldap://ldapmaster.ldap.local.site:9000)
+
+Attention: You _should_not_ edit the olc-Sections
+"config" and "ldif", as these are vital to the olc itself.
+
+
+b) MultiMaster-Configuration (MMR):
+At this time (S4 (pre)Alpha9) the only possible Replication setup.
+Use the provision-backend Parameter:
+
+ --ol-mmr-urls=<list of whitespace separated ldap-urls (and Ports <> 389!).
+
+e.g.:
+--ol-mmr-urls="ldap://ldapmaster1.ldap.local.site:9000 \ 
+   ldap://ldapmaster2.ldap.local.site:9000"
+
+Attention: You have to start OL with the commandline
+displayed in the output under (3), but you have to set a 
+listening port of slapd manually
+(e.g. -h ldap://ldapmaster1.ldap.local.site:9000)
+
+The Ports must be different from 389, as these are occupied by S4.
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+

selftest/target/Samba4.pm

@@ -840,7 +840,7 @@ sub provision($$$$$$$)
 		$ret->{LDAP_URI} = $ctx->{ldap_uri};
 		push (@{$ctx->{provision_options}},"--ldap-backend=$ctx->{ldap_uri}");
 
-		system("$self->{setupdir}/provision-backend $configuration --ldap-admin-pass=$ctx->{password} --root=$ctx->{unix_name} --realm=$ctx->{realm} --domain=$ctx->{domain} --host-name=$ctx->{netbiosname} --ldap-backend-type=$self->{ldap}>&2") == 0 or die("backend provision failed");
+                system("$self->{setupdir}/provision-backend $configuration --ldap-admin-pass=$ctx->{password} --root=$ctx->{unix_name} --realm=$ctx->{realm} --domain=$ctx->{domain} --host-name=$ctx->{netbiosname} --ldap-backend-type=$self->{ldap} --nosync>&2") == 0 or die("backend provision failed");
 
 		push (@{$ctx->{provision_options}}, "--password=$ctx->{password}");