README.4layout

(This file was generated by makeinfo and splitinfo.gawk.)
(Released under the old-style GNU documentation license;
 see sources or other output files for full text.)

6 Layout and infrastructure
***************************

The TeX Live source tree is the subtree rooted at 'Build/source' of the
complete TL distribution and contains the sources for all executables
distributed by TL, as well as 'configure' scripts and 'make' rules to
build and install them together with some of their support files.

6.1 Build system tools
======================

As mentioned above (*note Prerequisites::), a normal build requires very
little.  On the other hand, if you want to modify the TeX Live
infrastructure sources, such as 'configure.ac' or 'Makefile.am' files,
you will need to have several additional tools installed.

   In general, the TL build system uses the latest released versions of
the GNU build tools, installed directly from the original GNU releases
(e.g., by building them with 'configure --prefix=/usr/local/gnu' and
having 'PATH' start with '/usr/local/gnu/bin').  We have found that
trying to use the versions of these tools packaged for distros causes
many extra hassles, so don't do that, tempting as it may be.

   Currently the versions we use are:

     autoconf (GNU Autoconf) 2.69
     automake (GNU automake) 1.15.1
     bison (GNU Bison) 3.0.4
     flex 2.6.0
     ltmain.sh (GNU libtool) 2.4.6
     m4 (GNU M4) 1.4.18
     makeinfo (GNU texinfo) 6.5

   These versions should be used to update the generated files (e.g.,
'configure' or 'Makefile.in') in all or parts of the TL tree after their
dependencies have been changed.  This can be done explicitly with the
top-level 'reautoconf' script or implicitly by using the configure
option '--enable-maintainer-mode'.

   The files in the Subversion repository (see
<http://tug.org/texlive/svn>) are all up to date, but unfortunately this
may not be reflected by their timestamps.  (For starters, be sure to set
'use-commit-times=yes' in '~/.subversion/config' or the equivalent.)

   To avoid unnecessary runs of 'bison', 'flex', or 'makeinfo' it may be
necessary to 'touch' the generated ('.c', '.h', or '.info') files.  With
'--enable-maintainer-mode' it may also be necessary to 'touch' first
'aclocal.m4', then 'configure' and 'config.h.in' (or 'c-auto.in'), and
finally all 'Makefile.in' files.  Perhaps 'make -t' will help.

6.2 Top-level directories
=========================

Here is a brief description of the top-level directories in the TeX Live
source tree.

   As mentioned at the beginning of *note Overview of build system::,
the main source directories are 'texk/' (TeX-specific programs and
libraries), 'utils/' (additional programs), and 'libs/' (generic
libraries).

   In addition, the top-level directories 'am/' and 'm4/' contain
'Makefile.am' fragments and Autoconf macros, respectively, used in many
places.  Specifically, the file 'm4/kpse-pkgs.m4' contains lists of all
program and library modules; missing modules are silently ignored.
(This helps in creating cut-down source trees.)

   Each module contributes fragments (in separate files) defining its
capabilities and requirements to the 'configure.ac' scripts at the
top-level and in the subdirectories 'libs', 'utils', and 'texk'.  The
fragments from program modules supply 'configure' options to disable or
enable building them; those from library modules specify if an installed
(system) version of that library can be used.  This ultimately
determines which modules need to be built--although all modules must be
configured for the benefit of 'make' targets such as 'dist' or
'distcheck'.

   The top-level 'build-aux/' directory contains the common files
'compile', 'config.guess', 'config.sub', 'depcomp', etc. used by most
packages.  These are from the GNU Gnulib sources
(<http://www.gnu.org/software/gnulib>), which in turn synchronizes with
the appropriate ultimate upstream repository.  There are, however,
independent copies in, e.g., 'libs/freetype2/freetype-*/builds/unix/',
and a few other places.  The 'reautoconf' script does not touch those,
but a TL cron job keeps them in sync (nightly).

   The directory 'extra/' contains things which are not part of the TL
build, but are present just for (someone's) convenience, e.g., is
'epstopdf' development source is here.

   When the top-level './Build' script is used to build TL, two more two
more top-level directories appear: 'Work/' for the build tree, and
'inst/' for the install tree (from 'make install').  These names (and
everything else about 'Build''s operation) can be changed by setting
environment variables before running it; see the script file.

6.3 Autoconf macros
===================

Here we describe some of the Autoconf macros used in several modules-not
a complete list, by any means.  These general macros are supplemented by
module-specific macros in directories such as 'texk/dvipng/m4/'; some of
those are described in *note Library modules:: and *note Program
modules::.

6.3.1 General setup macros
--------------------------

The TL sources use two general setup macros:

 -- Macro: KPSE_BASIC (NAME, [MORE-OPTIONS])
     Initialize the basic TL infrastructure for module NAME:
        'AM_INIT_AUTOMAKE([foreign MORE-OPTIONS])'
        'AM_MAINTAINER_MODE'
        'KPSE_COMPILER_WARNINGS'
     and make sure the C compiler understands function prototypes.  This
     is used for all generic library and program modules.

 -- Macro: KPSE_COMMON (NAME, [MORE-OPTIONS])
     Like 'KPSE_BASIC' but add:
        'LT_PREREQ([2.2.6])'
        'LT_INIT([win32-dll])'
        'AC_SYS_LARGEFILE'
        'AC_FUNC_FSEEKO'
     and check for frequently used functions, headers, types, and
     structures.  This is used for TeX-specific modules.

6.3.2 Macros for programs
-------------------------

Macros for program checks:

 -- Macro: KPSE_CHECK_LATEX
     Set 'LATEX' to the name of the first of 'latex', 'elatex', or
     'lambda' which exists in 'PATH', or to 'no' if none of them exists.
     Call 'AC_SUBST' for 'LATEX'.  The result of this test can be
     overridden by setting the 'LATEX' environment variable or the cache
     variable 'ac_cv_prog_LATEX'.

 -- Macro: KPSE_CHECK_PDFLATEX
     Check for 'pdflatex' in 'PATH' and set 'PDFLATEX'.

 -- Macro: KPSE_CHECK_PERL
     Check for 'perl' or 'perl5' in 'PATH' and set 'PERL'.

 -- Macro: KPSE_PROG_LEX
     Call 'AC_PROG_LEX' and add the flag '-l' for 'flex'.

6.3.3 Macros for compilers
--------------------------

Macros for compiler-related checks:

 -- Macro: KPSE_COMPILER_WARNINGS
     When using the (Objective) C/C++ compiler, set
     'WARNING_[OBJ]C[XX]FLAGS' to suitable warning flags (depending on
     the value given to or implied for '--enable-compiler-warnings').
     Call 'AC_SUBST' for them.  At the moment this only works for GNU
     compilers, but could be extended to others if necessary.

     This macro caches its results in the 'kpse_cv_warning_cflags', ...
     variables.

 -- Macro: KPSE_COMPILER_VISIBILITY
     When using the C or C++ compiler, try to set
     'VISIBILITY_C[XX]FLAGS' to flags to hide external symbols.  Call
     'AC_SUBST' for this variable.  At the moment this only tests for
     the compiler option '-fvisibility=hidden', but that could be
     extended with more checks if necessary.

     This macro caches its results in the 'kpse_cv_visibility_cflags' or
     'kpse_cv_visibility_cxxflags' variable.

 -- Macro: KPSE_CXX_HACK
     Provide the configure option '--enable-cxx-runtime-hack'.  If
     enabled and when using 'g++', try to statically link with
     'libstdc++', somewhat improving portability of the resulting
     binary.

     This macro caches its result in the 'kpse_cv_cxx_hack' variable.

6.3.4 Macros for libraries
--------------------------

One macro for a library check:

 -- Macro: KPSE_LARGEFILE (VARIABLE, [EXTRA-DEFINE])
     Call 'AC_SYS_LARGEFILE' and 'AC_FUNC_FSEEKO' and append suitable
     '-D' flags (optionally including '-DEXTRA-DEFINE') to VARIABLE.

6.3.5 Macros for library and header flags
-----------------------------------------

Each library module 'libs/LIB' or 'texk/LIB' is supplemented by a macro
'KPSE_LIB_FLAGS' (all uppercase) that provides make variables for that
library.  E.g., for 'libs/libpng':

 -- Macro: KPSE_LIBPNG_FLAGS
     Provide the configure option '--with-system-libpng'.  Set and
     'AC_SUBST' 'make' variables for modules using this library (either
     an installed version or from the TeX Live tree): 'LIBPNG_INCLUDES'
     for use in 'CPPFLAGS', 'LIBPNG_LIBS' for use in 'LDADD',
     'LIBPNG_DEPEND' for use as dependency, and 'LIBPNG_RULE' defining
     'make' rules to rebuild the library.

 -- Macro: KPSE_ADD_FLAGS (NAME)
     Temporarily extend 'CPPFLAGS' and 'LIBS' with the values required
     for the library module 'NAME'.

 -- Macro: KPSE_RESTORE_FLAGS
     Restore 'CPPFLAGS' and 'LIBS' to their original values.

   As an example, the 'configure.ac' file for a hypothetical program
'utils/foo' using 'libpng', and hence 'zlib', would contain
     KPSE_ZLIB_FLAGS
     KPSE_LIBPNG_FLAGS

and its 'Makefile.am' would be along these lines:
     bin_PROGRAMS = foo
     AM_CPPFLAGS = ${LIBPNG_INCLUDES} ${ZLIB_INCLUDES}
     foo_LDADD = ${LIBPNG_LIBS} ${ZLIB_LIBS}
     foo_DEPENDENCIES = ${ZLIB_DEPEND} ${LIBPNG_DEPEND}
     ## Rebuild libz
     @ZLIB_RULE@
     ## Rebuild libpng
     @LIBPNG_RULE@

   If it was necessary to examine whether certain 'zlib' or 'libpng'
features were available, 'configure.ac' should be continued this way:
     KPSE_ADD_FLAGS([zlib])
     ... # tests for 'zlib' features, if any
     KPSE_ADD_FLAGS([libpng])
     ... # tests for 'libpng' features
     KPSE_RESTORE_FLAGS # restore 'CPPFLAGS' and 'LIBS'

6.3.6 Macros for Windows
------------------------

Windows differs in several aspects from Unix-like systems, many of them
due to the lack of symbolic links.

 -- Macro: KPSE_CHECK_WIN32
     Check if compiling for a Windows system.  The result is 'no' for
     Unix-like systems (including Cygwin), 'mingw32' for Windows with
     GCC, or 'native' for Windows with MSVC. The result is cached in the
     'kpse_cv_have_win32' variable.

 -- Macro: KPSE_COND_WIN32
     Call 'KPSE_CHECK_WIN32' and define the Automake conditional 'WIN32'
     ('true' if the value of 'kpse_cv_have_win32' is not 'no').

 -- Macro: KPSE_COND_MINGW32
     Call 'KPSE_COND_WIN32' and define the Automake conditional
     'MINGW32' ('true' if the value of 'kpse_cv_have_win32' is
     'mingw32').

 -- Macro: KPSE_COND_WIN32_WRAP
     Call 'KPSE_COND_WIN32' and define the Automake conditional
     'WIN32_WRAP' ('true' if the standard Windows wrapper
     ('texk/texlive/w32_wrapper/runscript.exe') exists.  This wrapper is
     used on Windows instead of symlinks for the "linked scripts" (*note
     Linked scripts::).

 -- Macro: KPSE_WIN32_CALL
     Call 'KPSE_COND_WIN32', check if the file
     'texk/texlive/w32_wrapper/callexe.c' exists; if it does, create a
     symlink in the build tree.  Compiling 'callexe.c' with
     '-DEXEPROG='"FOO.exe"'' and installing 'callexe.exe' as 'BAR.exe'
     is used on Windows instead of a symlink 'BAR->FOO' for Unix-like
     systems.

6.4 Library modules
===================

Here we discuss some specifics for a few of the libraries in TL, both
for the details themselves, and as a way of illuminating the general
structure and variation.

6.4.1 The 'png' library in 'libs/libpng'
----------------------------------------

This generic library uses the source tree in, e.g., the subdirectory
'libpng-src/' with all modifications for TL recorded in 'TLpatches/*'.
The 'configure.ac' fragment 'ac/withenable.ac' contains

     KPSE_WITH_LIB([libpng], [zlib])

specifying the module name, and indicating the dependency on 'zlib'.  A
third literal argument 'tree' would specify that the library from the
TeX Live tree cannot be replaced by a system version.  That not being
the case here, a second fragment 'ac/libpng.ac' contains

     KPSE_TRY_LIB([libpng],
                  [#include <png.h>],
                  [png_structp png; png_voidp io; png_rw_ptr fn;
     png_set_read_fn(png, io, fn);])

thus providing the simple C code

     #include <png.h>
     int main ()
     { png_structp png; png_voidp io; png_rw_ptr fn;
       png_set_read_fn(png, io, fn);
       return 0; }

which Autoconf uses to verify the usability of a system version with C
code.  The analogous macro 'KPSE_TRY_LIBXX' would check using C++ code.
These fragments are included by 'configure.ac' at the top level.

   For this library, among many other modules, a proxy build system for
TL is used ('configure.ac', 'Makefile.am', and 'include/Makefile.am'),
ignoring the distributed one.  Consequently, a few generated files and
auxiliary scripts are removed from the distributed source tree.  The
public headers 'png.h', 'pngconf.h', and 'pnglibconf.h' are "installed"
(as symlinks) under 'include/' in the build tree exactly as they are for
a system version under, e.g., '/usr/include/'.

   The module is supplemented by the file 'm4/kpse-libpng-flags.m4' that
defines the M4 macro 'KPSE_LIBPNG_FLAGS' used by all modules depending
on this library in their 'configure.ac' to generate the 'make' variables
'LIBPNG_INCLUDES' for use in 'CPPFLAGS', 'LIBPNG_LIBS' for use in
'LDADD', 'LIBPNG_DEPEND' for use as dependencies, and 'LIBPNG_RULE'
defining 'make' rules to rebuild the library.

   'm4/kpse-libpng-flags.m4' also supplies the configure option
'--with-system-libpng' and uses 'pkg-config' to determine the flags
required for the system library.

6.4.2 The 'zlib' library in 'libs/zlib'
---------------------------------------

This generic library is very much analogous to 'libpng', but without the
dependency on any other library.  The file 'm4/kpse-zlib-flags.m4'
supplies the configure option '--with-system-zlib', as well as
'--with-zlib-includes' and '--with-zlib-libdir' to specify non-standard
locations of the 'zlib' headers and/or library.

6.4.3 The 'freetype' library in 'libs/freetype2'
------------------------------------------------

This module uses a wrapper build system with an almost trivial
'configure.ac' and with a 'Makefile.am' that invokes 'configure' and
'make' for the distributed source, followed by 'make install' with the
build tree as destination.  The flags required for the system library
are obtained through 'freetype-config'.

6.4.4 The 'kpathsea' library in 'texk/kpathsea'
-----------------------------------------------

This is one of the TeX-specific libraries that are maintained as part of
TeX Live (*note (kpathsea)::).  Despite being a core part of the TeX
system, it is not a terribly special case in the infrastructure.  The
TeX libraries are Libtool libraries (static and/or shared) and are
installed by 'make install' together with the programs.  They are,
however, not part of the TL DVD as distributed by TeX user groups, and
have never been officially released for standalone use.

   It is possible, and perhaps even useful for distro builds (*note
Distro builds::), to specify the configure option
'--with-system-kpathsea' in order to use a system version of the
library.  Programs outside the TL tree should use 'pkg-config' for the
required flags.

   In addition to 'ac/withenable.ac' and 'ac/kpathsea.ac' there is a
third fragment 'ac/mktex.ac' included by both 'ac/withenable.ac' and
'configure.ac' that supplies configure options such as
'--enable-mktextfm-default', which determine the compile time default of
whether or not to run 'mktextfm' to generate a missing '.tfm' file.  In
any case, however, the command line options '-mktex=tfm' or
'-no-mktex=tfm' for the TeX-like engines override this default.

6.5 Program modules
===================

As with libraries (*note Library modules::), here we discuss the details
for a few of the programs in TL.

6.5.1 The 't1utils' package in 'utils/t1utils'
----------------------------------------------

Once again we use the distributed source tree 't1utils-src' with
modifications documented in 'TLpatches/*' and a proxy build system
consisting of 'configure.ac' and 'Makefile.am'.  The fragment
'ac/withenable.ac' contains

     KPSE_ENABLE_PROG([t1utils])

specifying the module name without any dependencies, and supplies the
configure option '--disable-t1utils'.

6.5.2 The 'xindy' package in 'utils/xindy'
------------------------------------------

This module uses the distributed source tree 'xindy-src/' with
modifications documented in 'TLpatches/*', a proxy 'configure.ac', and a
wrapper 'Makefile.am' that descends into 'xindy-src'.  The 'xindy' build
requires a 'make' that supports a 'VPATH' build, can handle all targets,
and do not refer to '${top_srcdir}' or '${top_builddir}'.  The fragment
'ac/withenable.ac' contains

     KPSE_ENABLE_PROG([xindy], , [disable native])
     m4_include(kpse_TL[utils/xindy/ac/xindy.ac])
     m4_include(kpse_TL[utils/xindy/ac/clisp.ac])

where 'disable' in the third argument indicates that 'xindy' is only
built if explicitly enabled by the user with 'configure --enable-xindy'
(the need for 'clisp' is too painful to require by default), and
'native' disallows cross compilation.  The additional fragments
'ac/xindy.ac' and 'ac/clisp.ac' specify more 'configure' options to be
seen at the top level with 'ac/xindy.ac' also included by
'configure.ac'.

6.5.3 The 'xdvik' package in 'texk/xdvik'
-----------------------------------------

This package is maintained as part of the TeX Live tree with sources in
its top level directory and the subdirectory 'gui'.  The fragment
'ac/withenable.ac' contains

     dnl extra_dirs = texk/xdvik/squeeze
     KPSE_ENABLE_PROG([xdvik], [kpathsea freetype2], [x])
     m4_include(kpse_TL[texk/xdvik/ac/xdvik.ac])

thus specifying the dependency on the 'kpathsea', 'freetype', and X11
libraries.  The M4 comment (following 'dnl') signals the subsidiary
'squeeze/configure.ac'.  This is needed because the main executable
'xdvi-bin' (to be installed as, e.g., 'xdvi-xaw') is for the 'host'
system whereas the auxiliary program 'squeeze/squeeze' has to run on the
'build' system and in a cross compilation they differ.

   The additional fragment 'ac/xdvik.ac' is also included by
'configure.ac' and supplies the configure option '--with-xdvi-x-toolkit'
also seen at the top-level.

6.5.4 The subdirectory 'utils/asymptote'
----------------------------------------

This subdirectory contains the sources for 'asy' and 'xasy' but due to
its complexity and prerequisites (e.g., OpenGL) it is not part of the TL
build system.  These programs must be built and installed independently,
but are included on the TL DVD together with their support files.

6.6 Extending TeX Live
======================

This section outlines the basic process for adding new packages to the
TL build system.

   In any case, a new package directory 'foo' should contain the
original sources, as modified for TL, in 'foo/foo-src', and the changes
should be documented in 'foo/TLpatches/*'; changes should also be
submitted upstream whenever reasonable, of course.  In addition, 'foo/'
will need the usual Automake build-related files ('configure.ac',
'Makefile.am', etc.  Please keep a 'ChangeLog' for all TL changes.

6.6.1 Adding a new program module
---------------------------------

A TeX-specific program module in a subdirectory 'texk/PROG' may use the
TeX-specific libraries and is included by adding its name 'PROG' to the
M4 list 'kpse_texk_pkgs' defined in 'm4/kpse-pkgs.m4'.

   A generic program module in a subdirectory 'utils/PROG' must not use
the TeX-specific libraries and is included by adding its name 'PROG' to
the M4 list 'kpse_utils_pkgs' in 'm4/kpse-pkgs.m4'.

   In either case, the subdirectory 'texk/PROG' or 'utils/PROG' must
provide a fragment 'ac/withenable.ac' that contains the M4 macro
'KPSE_ENABLE_PROG' defined in 'm4/kpse-setup.m4' with 'PROG' as the
mandatory first argument and three optional arguments:

  1. a list of required libraries from the TL tree;

  2. a list of options ('disable' if this module is not to be built
     without the configure option '--enable-PROG', 'native' if cross
     compilation is not possible, 'x' if the program requires X11
     libraries);

  3. a comment added to the help text for the 'configure' option
     '--enable-PROG' or '--disable-PROG'.

   If the module requires specific configure options to be seen at the
top level, they should be defined in an additional fragment 'ac/PROG.ac'
included from 'ac/withenable.ac' and 'configure.ac'.

   Usually, the new program is maintained somewhere outside of TeX Live.
In that case, we put the upstream sources into a subdirectory 'PROG-src'
(e.g., 'utils/newprog/newprog-src').  We do not run 'configure' in this
original '...-src' directory, only in our own directory, but we do
compile using the source files in '...-src'.

   So, these are the files that we must generally create:

'ac/withenable.ac'
     The 'KPSE_ENABLE_PROG' call just explained.

'configure.ac'
'Makefile.am'
     By merging the contents of the original 'configure.ac' (if
     provided) and a comparable program already in TL. There is no magic
     recipe, it's necessary to think about needs to be done in the
     original vs. in TeX Live.

'TLpatches/TL-Changes'
     Actions taken after getting the original source tree; typically
     removal of derived or unused common files.

'TLpatches/patch-...'
     If any changes are needed to the original sources, record the
     patches here so they can be applied next time.  And send them
     upstream so that perhaps they won't have to be.

'ChangeLog'
     Record all TL-specific changes, now and in the future.

   Then, run GNU 'autoreconf' in the new directory (*note Build system
tools::).  After that works, 'svn add' the necessary files, notably
'Makefile.in aclocal.m4 configure', and 'svn:ignore' the Automake cache
'autom4te.cache'.  (This is so people checking out the TL source tree do
not have to run any autotools, but can simply run 'configure'.)

   Then, run the TL tool 'reautoconf' in the top-level TL 'source/'
directory, to incorporate the new program into the build tree.

   Then, run (and rerun) a build until the program compiles and tests
successfully, probably involving rerunning autoreconf in the source
directory, 'make' in the build directory, etc.

   After final success, don't forget to commit.

6.6.2 Adding a new generic library module
-----------------------------------------

A generic library module in a subdirectory 'libs/LIB' must not depend on
TeX-specific libraries, by definition.  It is included by adding its
name 'LIB' to the M4 list 'kpse_libs_pkgs' in 'm4/kpse-pkgs.m4'--before
any other libraries from the TeX Live tree on which it depends.

   As with program modules, the subdirectory 'libs/LIB' must contain the
sources and build system for the library (and any installable support
programs) and a fragment 'ac/withenable.ac' that contains the M4 macro
'KPSE_WITH_LIB' defined in 'm4/kpse-setup.m4' with 'LIB' as the
mandatory first argument and two optional arguments: a list of required
libraries from the TL tree, and a list of options (currently there is
only one: specify 'tree' if this library cannot be replaced by a system
version).

   If a system version can be used, a second fragment 'ac/LIB.ac' is
needed, containing the M4 macro 'KPSE_TRY_LIB' (or 'KPSE_TRY_LIBXX')
with 'LIB' as the mandatory first argument and two additional arguments
for the Autoconf macro 'AC_LANG_PROGRAM' used to compile and link a
small C (or C++) program as sanity check for using the system library.

   In addition a file 'm4/kpse-LIB-flags' (at the top level) must define
the M4 macro 'KPSE_LIB_FLAGS' (all uppercase) setting up the 'make'
variables 'LIB_INCLUDES', 'LIB_LIBS', 'LIB_DEPEND', and 'LIB_RULE' with
the values required for 'CPPFLAGS', 'LDADD', dependencies, and a
(multi-line) 'make' rule to rebuild the library when necessary.  All of
that is needed for the library from the TL tree and, if supported, for a
system version.

   If a system library is allowed, 'KPSE_LIB_FLAGS' also provides the
configure option '--with-system-LIB' and uses the additional M4 macro
'KPSE_LIB_SYSTEM_FLAGS' to generate the 'make' variables for a system
library.  Furthermore the definition of the M4 macro
'KPSE_ALL_SYSTEM_FLAGS' in 'm4/kpse-pkgs.m4' must be extended by the
line:
   'AC_REQUIRE([KPSE_LIB_SYSTEM_FLAGS])'

6.6.3 Adding a new TeX-specific library module
----------------------------------------------

A TeX-specific library module in a subdirectory 'texk/LIB' may depend on
other TeX-specific libraries but must not depend on any generic library
from the TL tree.  It is included in the same general ways as a generic
library (see the previous section), with these modifications:

   * The library name 'LIB' is added to the M4 list 'kpse_texlibs_pkgs'
     also in 'm4/kpse-pkgs.m4'.

   * The fragment 'ac/withenable.ac' must use 'KPSE_WITH_TEXLIB'.