===============================================================================
                 Tips about Using Debian's MKL Package / FAQ
===============================================================================

                    -- Mo Zhou <cdluminate@gmail.com>, Last Update: May 11 2018

Table of Contents

  -- User Part --

  (*) How to install these MKL Debian packages?
  (*) How do I link against MKL?
  (*) Environment variables
  (*) MKL performance tips
  (*) Switching BLAS/LAPACK by environment variable and alternatives mechanism
  (*) Building custom library from MKL with selected symbols
  (*) Where are MKL documentations and examples?
  (*) Upstream's XXX is missing. Where is it?

  -- Debian Developer Part --

  (*) Is MKL really redistributable?
  (*) How do I obtain the upstream source tarball?
  (*) How do I understand the upstream MKL source directory structure?
  (*) How is this package organized?

-------------------------------------------------------------------------------
How to install these MKL Debian packages?
-------------------------------------------------------------------------------

(1) If you want the basic set (without cluster and multiarch) of MKL, which
    includes shared objects, static libraries and headers:

      $ sudo apt install intel-mkl

    For i386 architecture this is the top level metapackage.

(2) If you want the basic set of MKL together with cluster support, but without
    multiarch libs:

      $ sudo apt install intel-mkl-cluster

    This is only available on amd64.

(3) If you want all the components of MKL, i.e. the basic set, the cluster
    support and the multiarch libraries:

      $ sudo dpkg --add-architecture i386
      $ sudo apt update
      $ sudo apt upgrade
      $ sudo apt install intel-mkl-full

    This is only available on amd64. It's the top level metapackage for amd64.

(4) If you want to simply add a new alternative to libblas.so.3 and
    liblapack.so.3, without any header file:

      $ sudo apt install libmkl-rt

-------------------------------------------------------------------------------
How do I link against MKL?
-------------------------------------------------------------------------------

The simplest way to link against MKL is to link against libmkl_rt.so .
Just "-lmkl_rt". Or if you prefer pkg-config, compile program like this:

  $ g++ testdgemm.cc $(pkg-config --libs --cflags mkl-rt)

Here the mkl-rt.pc file is provided by Debian package instead of upstream.
The c++ source file mentioned in the above command line can be found under
the debian/tests directory.

For more complicated linking, refer any one of the following:

 * https://software.intel.com/en-us/mkl-linux-developer-guide-using-the-single-dynamic-library
 * intel-mkl-doc /usr/share/doc/intel-mkl/common/mkl_link_line_advisor.htm
 * intel-mkl-linktool:i386 (command line version of the above advisor)

-------------------------------------------------------------------------------
Environment variables
-------------------------------------------------------------------------------

You may need to manually export the NLSPATH variable, so that the catopen(3)
call is able to find the message catalog:

   $ export NLSPATH=/usr/share/locale/en_US/%N:$NLSPATH

Or $ export NLSPATH=/usr/share/locale/ja_JP/%N:$NLSPATH if your locale is ja_JP.
There is a file debian/tests/catalog.cc for testing catopen.

Apart from NLSPATH, the other environment variables prepared by upstream
scripts such as mklvars.sh are generally not needed.

-------------------------------------------------------------------------------
MKL performance tips
-------------------------------------------------------------------------------

Upstream's doc about performance and memory:

https://software.intel.com/en-us/mkl-linux-developer-guide-managing-performance-and-memory

Here are some points from package maintainer:

1. If your program computes weirdly slow with MKL, the following points
   may be helpful for you to find out the root of problem.

   * Set MKL_THREADING_LAYER=sequential environment variable if you
     are calling MKL from a multi-threading application.

   * Tweak environment variable OMP_NUM_THREADS or MKL_NUM_THREADS

   * Check if libgomp and libiomp5 are used at the same time.

   * For small size problems:
     https://software.intel.com/en-us/mkl-linux-developer-guide-improving-performance-for-small-size-problems

   * ...

   * Refer to upstream doc.

2. libmkl_rt.so is a magical run-time dispatcher which detects your
   CPU capacity at run time, and automatically selects shared objects
   from which symbols are loaded. By tracing the library calls you
   will figure out which shared objects are actually called.

     e.g. $ ltrace my_program

   See ltrace(1) for detail and advanced tracing.

-------------------------------------------------------------------------------
Switching BLAS/LAPACK by environment variable and alternatives mechanism
-------------------------------------------------------------------------------

Due to possible license violation, the default priority of MKL in the
alternatives mechanism is set to 1. See the following link for discussion:

  https://lists.debian.org/debian-science/2018/04/msg00071.html

However, if the user explicitly agrees to use MKL as the default BLAS/LAPACK,
libmkl_rt.so will be selected and be set in manual mode, overriding any
other alternatives. In contrast, we will make sure MKL is not selected
if the user rejects to use MKL as default.

(1) If you want to change the BLAS/LAPACK implementation, and you have root
    permission, the alternative mechanism is at your service.

    First let's check the current alternative status

      $ update-alternatives --get-selections | grep blas
      $ update-alternatives --get-selections | grep lapack

    Then, for example, let's switch the default BLAS implementation

      $ sudo update-alternatives --config libblas.so.3-${DEB_HOST_MULTIARCH}

    In the above command line, the actual value of placeholder
    "${DEB_HOST_MULTIARCH}" can be found by issuing this command

      $ dpkg-architecture -qDEB_HOST_MULTIARCH

    By the way, GUI frontends of update-alternatives such as galternatives
    will also work if you prefer them. For more information about usage
    of update-alternatives, see manual page update-alternatives(1).

(2) If you have NO root permission, but still want to change the default
    BLAS/LAPACK implementation for yourself, there are still solutions.
    Recall that as elaborated in manual page ls.so(8), both LD_LIBRARY_PATH
    and LD_PRELOAD can be used as solution for this purpose.
    Here is the LD_LIBRARY_PATH example:

      $ export LD_LIBRARY_PATH=usr/lib/${DEB_HOST_MULTIARCH}/mkl/

    The placeholder "${DEB_HOST_MULTIARCH}" is explained above. The above
    solution works because we've put two symbol links in that directory:

      usr/lib/${DEB_HOST_MULTIARCH}/mkl/libblas.so.3
      usr/lib/${DEB_HOST_MULTIARCH}/mkl/liblapack.so.3

    They are both symlinked to libmkl_rt.so . For detail see ld.so(8).

-------------------------------------------------------------------------------
Building custom library from MKL with selected symbols
-------------------------------------------------------------------------------

MKL provides a builder for building custom library from MKL.
The user can decide which symbol to include in this custom library.
After extracting upstream tarball, this builder locates here:

  opt/intel/compilers_and_libraries_2018.2.199/linux/mkl/tools/builder

In the Debian package that tool is moved to /usr/share/intel-mkl/ . Besides,
since Debian package doesn't retain the upstream directory structure, you
need to patch the builder so that it works correctly. However, due to intel's
ISSL license, we cannot install a patched builder for you. Here goes the
patch:

```
--- makefile.bak	2018-05-11 14:59:29.533147763 +0000
+++ makefile	2018-05-11 15:01:37.454016074 +0000
@@ -76,14 +76,11 @@
 
 ##------------------------------------------------------------------------------
 
-ifndef MKLROOT
-MKLROOT = ../..
-endif
-
-mklia32_libpath=$(MKLROOT)/lib/ia32
-mklintel64_libpath=$(MKLROOT)/lib/intel64
-compileria32_libpath=$(MKLROOT)/../compiler/lib/ia32
-compilerintel64_libpath=$(MKLROOT)/../compiler/lib/intel64
+MKLROOT = /usr
+mklia32_libpath=$(MKLROOT)/lib/i386-linux-gnu
+mklintel64_libpath=$(MKLROOT)/lib/x86_64-linux-gnu
+compileria32_libpath=$(MKLROOT)/lib/i386-linux-gnu
+compilerintel64_libpath=$(MKLROOT)/lib/x86_64-linux-gnu
 
 #ifndef export
 export=user_example_list
```

-------------------------------------------------------------------------------
Where are MKL documentations and examples?
-------------------------------------------------------------------------------

Install package 'intel-mkl-doc', and open this URL in your browser:

  file:///usr/share/doc/intel-mkl/ps2018/get_started.htm

This is an MKL documentation overview page. However network access
is required to browse further documents pointed by the page.

You can find some examples and miscellaneous files here after installing
the intel-mkl-doc package:

  /usr/share/doc/intel-mkl/

-------------------------------------------------------------------------------
Upstream's XXX is missing. Where is it?
-------------------------------------------------------------------------------

Indeed some files from upstream are not included in the Debian package because
they are troublesome or unusable due to some reason.

Contents not included in debian package:
  - libiomp5-related files
  - libtbb-related files
  - all benchmark files

If you need them, please download MKL tarball from Intel's site, or by apt:

  $ apt source intel-mkl

-------------------------------------------------------------------------------
Is MKL really redistributable?
-------------------------------------------------------------------------------

As declared by upstream at this page

  https://software.intel.com/en-us/mkl/license-faq
  "Yes, redistribution is allowed per the terms of the ISSL."

Even if registration is required before downloading the MKL tarball,
the tarball itself is re-distributable and licensed under ISSL.
See communication with upstream for more detail:

  https://github.com/intel/mkl-dnn/issues/206

-------------------------------------------------------------------------------
How do I obtain the upstream source tarball?
-------------------------------------------------------------------------------

Go to this page:

  https://software.intel.com/en-us/mkl

Click "Free Download". The tarball will be available after registration.
There are several hashsums in rules.

Note that, Intel provides MKL through other channels to, e.g. Pypi, APT, RPM.

-------------------------------------------------------------------------------
How do I understand the upstream MKL source directory structure?
-------------------------------------------------------------------------------

Go to the following page:

  https://software.intel.com/en-us/mkl-linux-developer-guide

I believe that section "Structure of the Intel Math Kernel Library" and
"Appendix C: Directory Structure in Detail" are exactly what you want.
Upstream documentation elaborates not only the directory structure, but
also individual files.

-------------------------------------------------------------------------------
How is this package organized?
-------------------------------------------------------------------------------

MKL package is a bit complex, but let me try to explain it.

Installation and Usage of this package are explained in previous sections
of this document, and they should have given you a basic idea how this package
is organized.

There is a detailed binary package index in debian/control. Files are carefully
split into small packages. It must be pointed out that the two most important
nodes in the dependency graph are libmkl-rt and libmkl-dev. Here are several
important notes about the dependency graph:

  * libmkl-rt is a run-time dispatching library. It auto matically selects
    shared object from which symbols are loaded. Typically it select one shared
    object from each of the three layers: interface layer, threading layer, and
    computational layer. To reduce my burden, I grouped the children librarires
    of libmkl-rt into three meta packages, respectively.

  * libmkl-rt plus the basic set of static libraries plus the headers equal
    to libmkl-dev.

  * intel-mkl metapackage is simply a wrapper of libmkl-dev. I expect that most
    user pull mkl by installing intel-mkl.

  * intel-mkl-cluster is intel-mkl + cluster support.

  * intel-mkl-full is intel-mkl + cluster support + multiarch (i386) libraries
    + all other misc stuff (e.g. custom library builder).

Despite of the 1000+ line control file, I tried my best to keep the rest part
of packaging simple and elegant. The python script debian/control.py generates
all the .install files and all the .lintian-overrides files. It scans upstream
tarball and put files in a proper package. I think the comments in the script
should be enough for one to fully understand what it does.

-------------------------------------------------------------------------------
TODO List
-------------------------------------------------------------------------------

* [cantfix] Performance on AMD CPUs. MKL looks specific to Intel CPUs.

* [wontfix] should we conflict with the MKL packages came from Intel's APT
  channel? I think there is no need to do that. Upstream .deb packages
  install stuff to /opt . There is no conflict.

* [wontfix] move headers out from directory /usr/include/mkl, to /usr/include.
  I think we'd better not move the headers to the public space. The MKL
  library ships fftw3 header, which is installed at

      /usr/include/mkl/fftw/fftw3.h

  And the above location won't cause any confusion, compared to

      /usr/include/fftw/fftw3.h  (from: libmkl-dev)
      /usr/include/fftw3.h       (from: libfftw3-dev)

  If we move the MKL headers out from a private directory.

* [wontfix] symbols control file. None of the shared object has SONAME.
  And dpkg-gensymbols won't generate symbols list. (dpkg-gensymbols)
