LD_LIBRARY_PATH considered harmful
The purpose of the
LD_LIBRARY_PATH environment variable is to instruct the
linker to consider additional directories when searching for libraries. Its
valid use case is the test of alternative library versions installed in
non-standard locations. In contrast to that, globally setting the
LD_LIBRARY_PATH (e.g. in the profile of a user) is harmful because there is
no setting that fits every program. The directories in the
environment variable are considered before the default ones and the ones
specified in the binary executable. Thus, a - say - system
command that is supposed to use a system library easily gets linked at runtime
with an API incompatible version. Also, having a program that relies on a
LD_LIBRARY_PATH setting creates the maintenance burden to always
accurately document that setting and distribute that documentation with the
binary. Instead, to avoid these issues, the additional directories (if any)
that should be searched by the runtime linker should be specified via linker
-R) at build-time. This results in those
directories being written to an ELF attribute that is considered by the runtime
linker (i.e. the runpath).
Usenet discussions about the miss-use of
LD_LIBRARY_PATH go back as early as
1993. In 1994, Casper H.s. Dik (who later posted as Sun engineer)
concludes his answer in
comp.unix.solaris with '
LD_LIBRARY_PATH: just say
no'. For context, the first Solaris version that comes with ELF executables and
shared libraries seems to be Solaris 2.0, which was released 1992. Linux
supports ELF since 1995.
Around 1999, David Barr published the article Why
LD_LIBRARY_PATH is bad.
It has two examples that detail how
LD_LIBRARY_PATH causes harm, motivates
valid uses and describes better alternative ways already available on Solaris
7. The page
LD_LIBRARY_PATH Is Not The Answer references David Barr's
article and calls globally setting the
LD_LIBRARY_PATH a 'complete hack'.
Also referenced by this page is Rod Evans' 2004 blog post
LD_LIBRARY_PATH - just
say no. He, as a Sun employee at that time - in his sun.com blog, details on
the sister variables
LD_LIBRARY_PATH_64 that are
also available on Solaris, in addition to
LD_LIBRARY_PATH. His acroread example shows
how they can complicate the situation such that even more harm is delivered.
His conclusion also is to use the runpath and where necessary to make use of
$ORIGIN linker variable - a variable that is substituted by the runtimee linker
with the path where the executable is located. Similar to this example
is the war story Purging
LD_LIBRARY_PATH written 2010 by Joseph D. Darcy
on his Oracle blog. He describes the 'messy' way the JDK used and manipulated
LD_LIBRARY_PATH until version 7. Again,
$ORIGIN is found to be a better
alternative mechanism for that use case. Another (then) Sun
colleague Ali Bahrami follows up on Evans with Avoiding
LD_LIBRARY_PATH: The Options, in 2007. He calls
'crude tool' and argues that it is probably the '#1 one way to
get yourself into trouble in an ELF environment'. As an
alternative he describes the
elfedit tool available in Solaris
11 and later Solaris 10 patch levels.
The Shared Library HowTo also references David Barr's article and concludes that it
is handy for development and testing, but shouldn't be modified by an installation process for normal use by normal users.
alternative, it includes an example how (on Linux) the runtime linker
/lib/ld-linux.so.2 can be explicitly invoked for executing a given binary
using an alternative search path.
The Sun Studio 12 Fortran Programming Guide (!) warns about
LD_LIBRARY_PATH for anything but test scenarios:
Use of the
LD_LIBRARY_PATHenvironment variable with production software is strongly discouraged. Although useful as a temporary mechanism for influencing the runtime linker’s search path, any dynamic executable that can reference this environment variable will have its search paths altered. You might see unexpected results or a degradation in performance.
Linux distributions usually don't include any package that relies on a certain
LD_LIBRARY_PATH setting - they install the packaged libraries into the
standard locations. But even a package distribution like OpenCSW (that
installs all its packages into a non-standard path) has a policy against
LD_LIBRARY_PATH for all the right reasons:
It is not necessary to set it for OpenCSW binaries. All of them are built with the -R flag, so each binary itself knows where to look for the shared objects.
You do not need to set
LD_LIBRARY_PATHsystem-wide; and if you do, you will likely break your system, even to the point of locking yourself out. Some of the library names clash between /usr/lib and
/opt/csw/lib, and if you run the Solaris openssh daemon with
/usr/lib/ssh/sshdwill try to load
/opt/csw/liband fail to start.
They also reference Rod Evan's blog article.
The title of this article is inspired by the considered harmful meme. See
for example Go To Statement Considered Harmful and Recursive Make
Considered Harmful. As with the
LD_LIBRARY_PATH that has legitimate uses,
goto has them as well, cf. Structured Programming with go to (Knuth,
Looking at the documented harmfulness of
LD_LIBRARY_PATH one might wonder why
it is popular in certain circles. One reason probably can be traced back to the
standard install note printed when installing a package that uses
Libtool (often used with Autoconf/Automake):
---------------------------------------------------------------------- Libraries have been installed in: $PREFIX/lib If you ever happen to want to link against installed libraries in a given directory, LIBDIR, you must either use libtool, and specify the full pathname of the library, or use the `-LLIBDIR' flag during linking and do at least one of the following: - add LIBDIR to the `LD_LIBRARY_PATH' environment variable during execution - add LIBDIR to the `LD_RUN_PATH' environment variable during linking - use the `-Wl,-rpath -Wl,LIBDIR' linker flag - have your system administrator add LIBDIR to `/etc/ld.so.conf' See any operating system documentation about shared libraries for more information, such as the ld(1) and ld.so(8) manual pages. ----------------------------------------------------------------------
This note contains two bad advices:
- the use of
- the use of
LD_RUN_PATH(which is in effect similar to
LD_LIBRARY_PATHbut only considered if the
-rpathoption isn't supplied)
It is unfortunate because the ramifications of the alternatives aren't qualified and
LD_LIBRARY_PATH is even mentioned first.
Thus, a developer or sysadmin who doesn't know much about linking might be
tempted to see the
LD_LIBRARY_PATH as THE standard way and because it works for
one package then wrongly internalize that as this-is-how-it-is-done-on-unix.
In addition to that, some vendors that distribute binary executables and libraries just give bad advice in their install instructions. For example Oracle, the well-known 'enterprise' DB vendor:
Add the name of the directory containing the Instant Client libraries to
(SQLPlus® User's Guide and Reference, Oracle 11g2, Configuring SQLPlus Instant Client)
Before you can connect Instant Client (including Instant Client Light) to an Oracle database, ensure that the
LD_LIBRARY_PATHenvironment variable specifies the directory that contains the Instant Client libraries.
(Database Client Installation Guide, Oracle 11g2, Recommended Postinstallation Tasks)
instantclient_12_1directory must be on the
LD_LIBRARY_PATHbefore linking the application.
(Oracle C++ Call Interface Programmer's Guide, Oracle 12c, Installation and Upgrading)
Last but not least, a quick google search regarding some
cannot-start-program-library-not-found error might turn up low quality forum posts,
where setting the
LD_LIBRARY_PATH is suggested.
Harm at compile time¶
At compile time, the linker
ld is usually called by the compiler such that
all object files a binary executable (or library) consists of are linked
together and dependent libraries are referenced. How
influences the linking differs on Linux and Solaris.
LD_LIBRARY_PATH directories aren't considered when
ld searches for
libraries specified via
-l. But, the
LD_LIBRARY_PATH is considered when
shared library dependencies of linked shared libraries are resolved (cf.
-rpath-link in ld(1)). In that case, the
are searched after the ones specified with
before ones specified by ELF attributes and the default ones (e.g.
On Solaris, in contrast to Linux, the
LD_LIBRARY_PATH directories are
ld when searching for libraries specified via
directories are appended to the search path resulting from any
Even if the
LD_LIBRARY_PATH is not globally set, it still may be in effect
because a poorly written makefile assigns this environment variable.
make is called from an IDE (like emacs) the environment of that
process is inherited - thus, an
LD_LIBRARY_PATH setting in the start script
of that IDE may induce harm.
Harm at runtime¶
At runtime, the runtime linker (e.g. on Linux this is
ld.so) searches the
LD_LIBRARY_PATH directories before the ones specified by the
DT_RUNPATH ELF attribute and the before default ones.
On Linux, the
DT_RPATH ELF attribute (which is documented as deprecated) is
considered before the
LD_LIBRARY_PATH, if and only if the binary doesn't also
DT_RUNPATH attribute set. In that case the
DT_RPATH ELF attribute
The writing of these two ELF attributes is system dependent:
|System||Compiler switch||ELF attribute|
- The compiler option
-Wlinstructs the compiler to pass the option following the first comma directly to the linker. All following commas are interpreted as argument delimiter.
- On Linux,
-Wl,-RSOMEDIRis equivalent due to option parsing magic - for compatibility reasons
-Ris overloaded. If the argument of
-Ris a filename the option has a different effect.
- On Solaris, the compiler and the linker both understand
-Wl,-Ris equivalent to
- The Solaris 10
-rpathalthough this isn't documented in all versions of the SunOS 5.10 ld(1) man page. It is documented in the 2011 version of that page, though.
Globally setting the
LD_LIBRARY_PATH is never a good idea. The
narrow original use case of
LD_LIBRARY_PATH are quick tests of alternate libraries.
When dealing with properly created executables setting the
LD_LIBRARY_PATH is redundant in the best case, but it breaks
things in the common case. There are better mechanisms and tools
LD_LIBRARY_PATH available to instruct the linker how to
search for the correct libraries at build-time and at runtime.
Verify Environment Settings¶
Verify that in fact
LD_LIBRARY_PATH (or it variants
isn't globally set via shell run control files like
/etc/bashrc or something like that. Also check
that it isn't set in user dotfiles like
etc. Such a setting would be bad in a config of a development user,
but exorbitantly more so in the profile of a production user.
If any running process still has the
LD_LIBRARY_PATH set can be
verified via looking at its environment. For example, on Linux
$ < /proc/$SOMEPID/environ tr '\0' '\n' | grep LD_LIBRARY_PATH
Or on Solaris via
$ pargs -e $SOMEPID | grep LD_LIBRARY_PATH
Set the runtime library search path at build time¶
Analogously to the
-LSOMEDIR option that adds a directory to the build-time
library search path, the option
-Wl,-RSOMEDIR) adds a
directory to the runtime library search path. That means that the
resulting path is written by the linker into the
DT_RUNPATH ELF attribute of the resulting binary.
The thus set attributes can be printed on Linux via
$ readelf -d my_binary_or_so | grep PATH 0x000000000000001d (RUNPATH) Library runpath: [SOMEDIR]
And on Solaris via
$ elfdump my_binary_or_so | grep PATH  RUNPATH 0x128 SOMEDIR  RPATH 0x128 SOMEDIR
Other interesting attributes dumped by those tools and that are relevant in
this context are
NEEDED (i.e. the dependent shared libraries specified via
-l or as absolute path) and
SONAME (i.e. the name of a shared library
that is copied from the library to the
NEEDED attribute of the binary that
depends on that library).
The effect of different runpaths and specified libraries can be verified via calling
ldd my_binary_or_so. When it outputs contains lines like
libxyz.so => not found
on Linux, or on Solaris
libxyz.so => (file not found)
then the path is still incomplete or incorrect and the runtime linker will abort the program start with a message like this
./my_binary: error while loading shared libraries: libxyz.so: cannot open shared object file: No such file or directory
on Linux and on Solaris:
ld.so.1: my_binary: fatal: libxyz.so: open failed: No such file or directory Killed
The Linux runtime linker exits with exit status 127, while the Solaris runtime linker exits with 137.
When the start of a binary has succeeded one can verify the actually runtime linked libraries via
pldd, which is available on Linux and Solaris. Or, as an alternative, via
The linker variable
$ORIGIN is expanded by the runtime linker
with the current 'origin' of the ELF binary.
The origin is the directory where the binary is stored.
Thus, using this variable in a path specified via
-Wl,-rpath,SOMEDIR (or via
-Wl,-RSOMEDIR) allows for path
specifications that are relative to the location of the binary.
The obvious usecase are binaries that are supposed to be installed inside a non-standard prefix (e.g.
/opt/foo) with some of its needed libraries. The directory could be then specified like this (in a shell):
Note that the linker variable
$ORIGIN is enclosed in single
quotes so that it is not expanded by the shell (e.g. bash).
When using this in a makefile, in addition to the single quotes, the dollar sign has to be escaped such that make doesn't expand it as make variable:
The linker variable
$ORIGIN is understood by the Linux and by
the Solaris runtime linker.
On Linux, the runtime linker also expands a few other variables.
On Solaris, use
-Wl,-i instructs the linker to ignore any
LD_LIBRARY_PATH environment variable. Thus, this variable can
be used as safety net in case
LD_LIBRARY_PATH accidentally is
ld on Linux interprets
(i.e. as: link incrementally).
Patch existing ELF binaries¶
In case one doesn't have access to the source code, it is still
an option to rewrite the
of an exisiting ELF binary. The tool
patchelf supports this.
For example, to fix some Oracle executables and the client library that are part of the Oracle 11g2 'Instant Client' (for Linux):
$ patchelf --set-rpath '$ORIGIN/..' /path/to/instantclient_11_2/sdk/proc $ patchelf --set-rpath '$ORIGIN' /path/to/instantclient_11_2/sqlplus $ patchelf --set-rpath '$ORIGIN' /path/to/instantclient_11_2/libclntsh.so.11.1
The effectiveness of such changes can be verified with the usual tools, e.g.:
$ patchelf --print-rpath mybinary # or: $ readelf -d mybinary | grep PATH $ ldd mybinary
After the change the
ldd utility shouldn't print any 'not found' lines, anymore.
Patchelf is packaged for the major Linux distributions and should also be portable to other ELF platforms.
Solaris 11 (and apparently later Solaris 10 patch levels) come
with the tool
elfedit that can also be used to edit the
DT_RPATH ELF attributes. Example:
$ elfedit -e 'dyn:runpath $ORIGIN/lib' mybinary
However (in contrast to patchelf) it has some limitations
(cf. its man page or Changing ELF Runpaths) - e.g. such
that elfedit doesn't find enough space for path edits. Especially
with binaries created on previous Solaris 10 (or even older
Solaris) versions this is issue. Later versions reserve some
space (512 bytes it seems) at build-time - such that the room for edits is of fixed size.
Thus, it is easy to construct a path that
patchelf has no issue
to add but where
elfedit fails with:
elfedit: [0: .dynstr]: String table does not have room to add string
Also, the dependency management of Solaris 10 doesn't seem very
complete such that a system may provide
elfedit but still miss
some libraries for it:
ld.so.1: elfedit: fatal: liblddbg.so.4: version 'SUNWprivate_4.83' not found (required by file /usr/bin/elfedit) ld.so.1: elfedit: fatal: liblddbg.so.4: open failed: No such file or directory
Obviously, when dealing with such poorly created binaries, created by an overpaid vendor, one may see this as indicator of the general quality of the provided software and service. And perhaps one reaches to the conclusion that there are better alternatives out there, built by people who know what they are doing. For our initial Oracle example the obvious alternative would be PostgreSQL. It is arguably of better quality than Oracle, implements features Oracle doesn't have and it is ridiculously easy to install (in comparison to Oracle) because it is available from the distributions package repositories.
As a last resort, when re-linking or patching an existing ELF binary is
not an option one should at least restrict the scope of
LD_LIBRARY_PATH to that binary, i.e. to a start script of that
For example, if the original legacy binary is located under
/opt/sware/bin/foo one limits the harm of
putting it in a start script like this:
$ mv /opt/sware/bin/foo /opt/sware/bin/foo.orig $ cat <<EOF > /opt/sware/bin/foo #!/bin/sh export LD_LIBRARY_PATH=/opt/sware/lib exec /opt/sware/bin/foo.orig "$@" EOF $ chmod 755 /opt/sware/bin/foo
Thus, its effect is limited to the legacy binary. This is a significant improvement over globally setting it.
In case the process forks any child processes, the
LD_LIBRARY_PATH setting is inherited, though.
This can be avoided via directly invoking the runtime linker and supplying the search path as an argument. A Linux example:
$ mv /opt/sware/bin/foo /opt/sware/bin/foo.orig $ cat <<EOF > /opt/sware/bin/foo #!/bin/sh exec /lib64/ld-linux-x86-64.so.2 --library-path /opt/sware/lib \ /opt/sware/bin/foo.orig "$@" EOF $ chmod 755 /opt/sware/bin/foo
This the runtime linker for a 64 bit binary, for a 32 bit binary
one would use
A Solaris example:
$ mv /opt/sware/bin/foo /opt/sware/bin/foo.orig $ cat <<EOF > /opt/sware/bin/foo #!/bin/sh exec /lib/64/ld.so.1 -e LD_LIBRARY_PATH=/opt/sware/lib \ /opt/sware/bin/foo.orig "$@" EOF $ chmod 755 /opt/sware/bin/foo