some issues with the proposals for the python packaging in..

While preparing some example packages to experiment with
python-central and python-support, I did see some issues with both
proposals, in that the dependencies are not fulfilled for every python
version that both packaging systems claim to support. Feedback is
welcome.

For an example see python-pmw (only one binary-all package with the
same name is built by the source package):

Package: python-pmw
Depends: python-tk, python (>= 2.3), python (<< 2.4)

which, when packaged with one of the packaging systems, becomes:

Package: python-pmw
Depends: python-tk, python (>= 2.3)

Trying to use python-pmw with a python version, which is not the
default, will fail, if the pythonX.Y-tk package is not installed for
that version. To generalize, every binary-all python library package
depending on a binary-arch package (containing an extension module)
does have this problem.

Looking at an application using python-pmw (i.e. pymol):

Package: pymol
Depends: python (>= 2.3), python-pmw

The package will still work after an upgrade of the default python
version. Assuming that an application package uses a specific python
version, i.e.:

Depends: ${python:Depends}, python-pmw
-->
Depends: python2.3, python-pmw

the package dependencies may not be fulfilled anymore. That could be
solved by adding to python:Depends python2.3-tk. Either the package
maintainer has to do that explicitely, or dh_python has to find these.

The issues (and questions) are:

- The packaging infrastructure forces the installation of the default
python version, it's not possible to install a non-default version on
it's own (if at least one package using the infrastructure is
installed).
At least thats one thing I can live with; others as well?

- As outlined above, we cannot enforce correct dependencies with the
proposed packaging infrastructure (both variants). That is only the
case when using a non-default python version. AFAICS this would
violate the Debian policy. Should there be an exception?

- A packaging infrastructure not supporting binary-arch modules
covers about 50 out of 200 source packages providing python modules
and extensions (that number may not be accurate, just counting
packages using the python- and pythonX.Y- prefixes).

That number can be raised, if extension modules for all supported
python versions are made available in one package (at least for the
version we transition from and for the version we transition to).
This approach has it's limitations, i.e. python2.3-tk and
python2.4-tk are built from separate sources and cannot be put in
one binary package. It does help for packages like zopeinterface
and twisted, where only very small extension modules are put in
one package supporting more than one python version. Even larger
extension modules could be packages this way for at least the
time of a python transition to support both old and new version (a
package like pygtk2 comes to mind, having many rdepends).

We still do have the limitation, that every python module depending
on a pythonX.Y-foo binary-arch package cannot use the packaging
infrastructure.

- AFAICS the proposed packaging infrastructure doesn't help the
migration of a new python default version to testing much. It does
help maintainers of these 50 source packages, but still requires
uploads of the other 150 packages (potentially introducing
dependencies on newly uploaded libs). Supporting more than one
python version for binary-arch packages does raise that number.

- Just another proposal could be to keep the package structure
python-foo, python2.3-foo, python2.4-foo, put all arch independent
files into python-foo, using the proposed python infratstructure
to promote the packages to each python version and putting
extension modules into the pythonX.Y packages. In case of
binary-all modules, the pythonX.Y packages are just dependency
packages.
That proposal does address the concern of putting the source in
only one package, avoiding code dubplication in binary packages,
but still requires new upload of the package for a python
transition, not supporting a migration to testing very well.
There are some packages using such kind of setup.

Can we neglect the dependency issues for modules available for
non-default python versions, seeing these just as an aid for doing a
transition and require packages explicitely using a non-default
version to add these dependencies themself?


Matthias


Appendix:

Source packages, building packages with at least one binary-any dep: (146)

apoo, avahi, babel, celementtree, cheetah, clearsilver,
codespeak-lib, configlet, ctypes, dbus, diacanvas2, dnspython,
egenix-mx-base, elementtree, eunuchs, eyed3, file, forgethtml,
forgetsql, gadfly, gdal, gnome-python, gnome-python-extras,
gnuradio-core, gpib, gst-python, jabber.py, ldaptor,
libmetakit2.4.9.3, libmusicbrainz-2.1, libtunepimp, libxml2,
libxslt, lxml, matplotlib, maxdb-7.5.00, nautilus-python, nevow,
paramiko, plplot, poker-engine, poker-network, psyco, psycopg,
pycairo, pyfribidi, pygame, pygdchart2, pygresql, pygtk, pygtkmvc,
pylint, pymad, pyopengl, pyopenssl, pyorbit, pysvn, pytables,
python-4suite, python-adns, python-apsw, python-apt,
python-biggles, python-biopython, python-bsddb3, python-cdd,
python-cjkcodecs, python-crack, python-crypto, python-davlib,
python-defaults, python-docutils, python-extclass, python-f2py,
python- fam, python-fuse, python-gnome, python-gnome (1.4.5-4),
python-gnuplot, python-imaging, python-japanese-codecs,
python-kde3, python-kinterbasdb, python-ldap, python-mysqldb,
python-numarray, python-numeric, python-omniorb2, python-orbit,
python-osd, python-oss, python-pam, python-pgsql, python-pmw,
python-pychart, python-pylibacl, python-pysqlite1.1,
python-pysqlite2, python-pyxattr, python-qt3, python-reportlab,
python-scientific, python-scipy, python-scipy-core, python-simpy,
python-soappy, python-sqlite, python-tclink, python-tcpwrap,
python-unit, python-visual, python-xml, python-xmpp, python2.4,
pythoncad, pythoncard, pyvorbis, pyxmms, pyxmpp, rpy, rrdtool,
schoolbell, schooltool, simpleparse, sip-qt3, sip4-qt3, soya,
sqlrelay, syck, twisted, twisted-conch, twisted-lore,
twisted-mail, twisted-names, twisted-news, twisted-runner,
twisted-web, twisted-web2, utidylib, vte, wxglade, wxwindows2.4,
wxwidgets2.6, xmldiff, zopeinterface, zsi


Source package, building packages with only binary-all deps: (47)

albatross, beautifulsoup, celementtree, cherrypy2.1, clientcookie,
constraint, elementtree, empy, foomatic-gui, htmlgen, ipy,
jabber.py, logilab-astng, logilab-common, moin, optcomplete,
pexpect, pmock, py-libmpdclient, pycxx, pyparsing, pyrex, pyspf,
python-4suite, python-adodb, python-cherrypy python-dhm,
python-docutils, python-goopy, python-htmltmpl, python-iplib,
python-irclib, python-medusa, python-pyrss2gen, python-pysnmp2,
python-setuptools, python-simpy, python-tz, python-weblib,
python-xlib, python-xmpp, pyvtk, simpletal, slides, sqlobject,
yappy

Removed fixedpoint (only needed for python < 2.3)


--
To UNSUBSCRIBE, email to debian-python-REQUEST.DeleteThis@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster.DeleteThis@lists.debian.org

Le jeudi 26 janvier 2006 à 15:26 +0100, Matthias Klose a écrit :
> While preparing some example packages to experiment with
> python-central and python-support, I did see some issues with both
> proposals, in that the dependencies are not fulfilled for every python
> version that both packaging systems claim to support. Feedback is
> welcome.
>
> For an example see python-pmw (only one binary-all package with the
> same name is built by the source package):
>
> Package: python-pmw
> Depends: python-tk, python (>= 2.3), python (<< 2.4)
>
> which, when packaged with one of the packaging systems, becomes:
>
> Package: python-pmw
> Depends: python-tk, python (>= 2.3)
>
> Trying to use python-pmw with a python version, which is not the
> default, will fail, if the pythonX.Y-tk package is not installed for
> that version. To generalize, every binary-all python library package
> depending on a binary-arch package (containing an extension module)
> does have this problem.

This is indeed a very problematic issue, and such a case will always
fail by design with all implementations that have been proposed.

[snip]

> - The packaging infrastructure forces the installation of the default
> python version, it's not possible to install a non-default version on
> it's own (if at least one package using the infrastructure is
> installed).
> At least thats one thing I can live with; others as well?

I don't think it's a big issue either.

> - As outlined above, we cannot enforce correct dependencies with the
> proposed packaging infrastructure (both variants). That is only the
> case when using a non-default python version. AFAICS this would
> violate the Debian policy. Should there be an exception?

This would become a nightmare for maintainers. For example, when needing
to use python2.4-pmw, the maintainer would have to check by himself that
he needs to depend on:
python2.4, python-pmw, python2.4-tk
(^^^^^^ or python2.4-pmw if there's a Provides:)

This makes dependencies look inconsistent and it makes impossible to
easily distinguish which packages don't have correct dependencies.
That's too high a price to pay for only a small part of public modules.

> - A packaging infrastructure not supporting binary-arch modules
> covers about 50 out of 200 source packages providing python modules
> and extensions (that number may not be accurate, just counting
> packages using the python- and pythonX.Y- prefixes).
>
> That number can be raised, if extension modules for all supported
> python versions are made available in one package (at least for the
> version we transition from and for the version we transition to).
> This approach has it's limitations, i.e. python2.3-tk and
> python2.4-tk are built from separate sources and cannot be put in
> one binary package. It does help for packages like zopeinterface
> and twisted, where only very small extension modules are put in
> one package supporting more than one python version. Even larger
> extension modules could be packages this way for at least the
> time of a python transition to support both old and new version (a
> package like pygtk2 comes to mind, having many rdepends).
>
> We still do have the limitation, that every python module depending
> on a pythonX.Y-foo binary-arch package cannot use the packaging
> infrastructure.

I really believe mixing binary-arch modules for several python versions
is a slippery slope. We should avoid such horrors whenever possible.

> - AFAICS the proposed packaging infrastructure doesn't help the
> migration of a new python default version to testing much. It does
> help maintainers of these 50 source packages, but still requires
> uploads of the other 150 packages (potentially introducing
> dependencies on newly uploaded libs). Supporting more than one
> python version for binary-arch packages does raise that number.

But makes dependency management a nightmare.

> - Just another proposal could be to keep the package structure
> python-foo, python2.3-foo, python2.4-foo, put all arch independent
> files into python-foo, using the proposed python infratstructure
> to promote the packages to each python version and putting
> extension modules into the pythonX.Y packages. In case of
> binary-all modules, the pythonX.Y packages are just dependency
> packages.
> That proposal does address the concern of putting the source in
> only one package, avoiding code dubplication in binary packages,
> but still requires new upload of the package for a python
> transition, not supporting a migration to testing very well.
> There are some packages using such kind of setup.

But again, they make dependencies look inconsistent. In this case,
depending on pythonX.Y-foo isn't sufficient to have the foo module
available.

> Can we neglect the dependency issues for modules available for
> non-default python versions, seeing these just as an aid for doing a
> transition and require packages explicitely using a non-default
> version to add these dependencies themself?

I don't think so. This would sacrifice the packaging simplicity and
clean dependencies on the altar of the testing scripts. While it is a
good idea to help testing scripts whenever possible, it shouldn't be a
goal by itself.

After this new input and some thinking, I'm afraid that none of the
implementations that have been proposed (including the one I wrote) look
satisfactory for public modules, whether they be binary or python-only
modules.

There is still a situation we can improve easily, though: private
modules. Currently, they have to migrate with python transitions, and
this is only because of byte-compilation. The python-support way of
doing things should still be fine for them, and it can reduce the number
of packages to migrate at once, without complicating anyone's work.
--
.''`. Josselin Mouette /\./\
: :' : josselin.mouette.RemoveThis@ens-lyon.org
`. `' joss.RemoveThis@debian.org
`- Debian GNU/Linux -- The power of freedom