%feature("docstring") OT::GeneralLinearModelAlgorithm
"Algorithm for the evaluation of general linear models.

Available constructors:
    GeneralLinearModelAlgorithm(*inputSample, outputSample, covarianceModel, basis, normalize=True, keepCovariance=True*)

    GeneralLinearModelAlgorithm(*inputSample, inputTransformation, outputSample, covarianceModel, basis, keepCovariance=True*)

    GeneralLinearModelAlgorithm(*inputSample, outputSample, covarianceModel, basisCollection, normalize=True, keepCovariance=True*)

    GeneralLinearModelAlgorithm(*inputSample, inputTransformation, outputSample, covarianceModel, basisCollection, keepCovariance=True*)

Parameters
----------
inputSample, outputSample : :class:`~openturns.Sample` or 2d-array
    The samples :math:`(\vect{x}_k)_{1 \leq k \leq N} \in \Rset^n` and :math:`(\vect{y}_k)_{1 \leq k \leq N}\in \Rset^d`.
inputTransformation : :class:`~openturns.Function`
    Function :math:`T` used to normalize the input sample.

    If used, the meta model is built on the transformed data.
basis : :class:`~openturns.Basis`
    Functional basis to estimate the trend: :math:`(\varphi_j)_{1 \leq j \leq n_1}: \Rset^n \rightarrow \Rset`. 

    If :math:`d>1`, the same basis is used for each marginal output.
basisCollection : collection of :class:`~openturns.Basis`
    Collection of :math:`d` functional basis: one basis for each marginal output.

    An empty collection means that no trend is estimated.
covarianceModel : :class:`~openturns.CovarianceModel`
    Covariance model of the Gaussian process. See notes for the details.
normalize : bool, optional
    Indicates whether the input sample has to be normalized.

    OpenTURNS uses the transformation fixed by the User in *inputTransformation* or the empirical mean and variance of the input sample.
    Default is set in resource map key `GeneralLinearModelAlgorithm-NormalizeData`
keepCovariance : bool, optional
    Indicates whether the covariance matrix has to be stored in the result structure *GeneralLinearModelResult*.
    Default is set in resource map key `GeneralLinearModelAlgorithm-KeepCovariance`

Notes
-----
We suppose we have a sample :math:`(\vect{x}_k, \vect{y}_k)_{1 \leq k \leq N}` where :math:`\vect{y}_k = \cM(\vect{x}_k)` for all :math:`k`, with :math:`\cM:\Rset^n \mapsto \Rset^d` a given function.

The objective is to build a metamodel :math:`\tilde{\cM}`, using a **generalized linear model**: the sample :math:`(\vect{y}_k)_{1 \leq k \leq N}` is considered as the restriction of a Gaussian process :math:`\vect{Y}(\omega, \vect{x})` on :math:`(\vect{x}_k)_{1 \leq k \leq N}`. The Gaussian process :math:`\vect{Y}(\omega, \vect{x})` is defined by:

.. math::

    \vect{Y}(\omega, \vect{x}) = \vect{\mu}(\vect{x}) + \vect{W}(\omega, \vect{x})

where:

.. math::

    \vect{\mu}(\vect{x}) = \left(
      \begin{array}{l}
        \mu_1(\vect{x}) \\
        \dots  \\
        \mu_d(\vect{x}) 
       \end{array}
     \right)

with :math:`\mu_\ell(\vect{x}) = \sum_{j=1}^{n_\ell} \beta_j^\ell \varphi_j^\ell(\vect{x})` and :math:`\varphi_j^\ell: \Rset^n \rightarrow \Rset` the trend functions.

:math:`\vect{W}` is a Gaussian process of dimension :math:`d` with zero mean and covariance function :math:`C = C(\vect{\theta}, \vect{\sigma}, \mat{R}, \vect{\lambda})` (see :class:`~openturns.CovarianceModel` for the notations).

We note:

.. math::

    \vect{\beta}^\ell = \left(
      \begin{array}{l}
        \beta_1^\ell \\
        \dots  \\
        \beta_{n_\ell}^\ell
       \end{array}
     \right) \in \Rset^{n_\ell}
     \quad \mbox{ and } \quad 
     \vect{\beta} = \left(
      \begin{array}{l}
         \vect{\beta}^1\\
        \dots  \\
         \vect{\beta}^d
       \end{array}
     \right)\in \Rset^{\sum_{\ell=1}^p n_\ell}


The *GeneralLinearModelAlgorithm* class estimates the coefficients :math:`\beta_j^\ell, \vect{p}` where :math:`\vect{p}` is the vector of parameters of the covariance model (a subset of :math:`\vect{\theta}, \vect{\sigma}, \mat{R}, \vect{\lambda}`) that has been declared as *active* (by default, the full vectors :math:`\vect{\theta}` and :math:`\vect{\sigma}`).

The estimation is done by maximizing the *reduced* log-likelihood of the model, see its expression below.

If a normalizing transformation :math:`T` has been used, the meta model is built on the inputs :math:`\vect{z}_k = T(\vect{x}_k)`.

**Estimation of the parameters** :math:`\beta_j^\ell, \vect{p}`

We note:

.. math::

    \vect{y} = \left(
      \begin{array}{l}
        \vect{y}_1 \\
        \dots  \\
        \vect{y}_N
       \end{array}
     \right) \in \Rset^{dN},
     \quad 
     \vect{m}_{\vect{\beta}} = \left(
      \begin{array}{l}
        \vect{\mu}(\vect{x}_1) \\
        \dots  \\
        \vect{\mu}(\vect{x}_N)
       \end{array}
     \right) \in \Rset^{dN}

 and 

.. math::

    \mat{C}_{\vect{p}} = \left(
      \begin{array}{lcl}
        \mat{C}_{11} & \dots &  \mat{C}_{1N}\\
        \dots & \dots & \\
        \mat{C}_{N1} & \dots &  \mat{C}_{NN}
       \end{array}
     \right) \in \cS_{dN}^+(\Rset)

where :math:`\mat{C}_{ij} = C_{\vect{p}}(\vect{x}_i, \vect{x}_j)`. 

The model likelihood writes:

.. math::

    \cL(\vect{\beta}, \vect{p};(\vect{x}_k, \vect{y}_k)_{1 \leq k \leq N}) = \dfrac{1}{(2\pi)^{dN/2} |\det \mat{C}_{\vect{p}}|^{1/2}} \exp\left[ -\dfrac{1}{2}\Tr{\left( \vect{y}-\vect{m} \right)} \mat{C}_{\vect{p}}^{-1}  \left( \vect{y}-\vect{m} \right)  \right]

If :math:`\mat{L}` is the Cholesky factor of :math:`\mat{C}`, ie the lower triangular matrix with positive diagonal such that :math:`\mat{L}\,\Tr{\mat{L}} = \mat{C}`, then:

.. math::
    :label: logLikelihood

    \log \cL(\vect{\beta}, \vect{p};(\vect{x}_k, \vect{y}_k)_{1 \leq k \leq N}) = cste - \log \det \mat{L}_{\vect{p}} -\dfrac{1}{2}  \| \mat{L}_{\vect{p}}^{-1}(\vect{y}-\vect{m}_{\vect{\beta}}) \|^2

The maximization of :eq:`logLikelihood` leads to the following optimality condition for $\vect{\beta}$:

.. math::

    \vect{\beta}^*(\vect{p}^*)=\argmin_{\vect{\beta}} \| \mat{L}_{\vect{p}^*}^{-1}(\vect{y}-\vect{m}_{\vect{\beta}}) \|^2

This expression of :math:`\vect{\beta}^*` as a function of :math:`\vect{p}^*` is taken as a general relation between :math:`\vect{\beta}` and :math:`_vect{p}` and is substituted into :eq:`logLikelihood`, leading to a *reduced log-likelihood* function depending solely on :math:`\vect{p}`.

In the particular case where :math:`d=\dim(\vect{\sigma})=1` and :math:`\sigma` is a part of :math:`\vect{p}`, then a further reduction is possible. In this case, if :math:`\vect{q}` is the vector :math:`\vect{p}` in which :math:`\sigma` has been substituted by 1, then:

.. math::

    \| \mat{L}_{\vect{p}}^{-1}(\vect{y}-\vect{m}_{\vect{\beta}}) \|^2=\| \mat{L}_{\vect{q}}^{-1}(\vect{y}-\vect{m}_{\vect{\beta}}) \|^2/\sigma^2

showing that :math:`\vect{\beta}^*` is a function of :math:`\vect{q}^*` only, and the optimality condition for :math:`\sigma` reads:

.. math::

    \vect{\sigma}^*(\vect{q}^*)=\dfrac{1}{N}\| \mat{L}_{\vect{q}^*}^{-1}(\vect{y}-\vect{m}_{\vect{\beta}^*(\vect{q}^*)}) \|^2
   
which leads to a further reduction of the log-likelihood function where both :math:`\vect{\beta}` and :math:`\sigma` are replaced by their expression in terms of :math:`\vect{q}`.
    
The default optimizer is :class:`~openturns.TNC` and can be changed thanks to the *setOptimizationAlgorithm* method.
User could also change the default optimization solver by setting the `GeneralLinearModelAlgorithm-DefaultOptimizationAlgorithm` resource map key to one of the :class:`~openturns.NLopt` solver names.

It is also possible to proceed as follows:

    - ask for the reduced log-likelihood function of the *GeneralLinearModelAlgorithm* thanks to the *getObjectiveFunction()* method
    - optimize it with respect to the parameters :math:`\vect{\theta}` and  :math:`\vect{\sigma}` using any optimisation algorithms (that can take into account some additional constraints if needed)
    - set the optimal parameter value into the covariance model used in the *GeneralLinearModelAlgorithm*
    - tell the algorithm not to optimize the parameter using *setOptimizeParameter*

The behaviour of the reduction is controlled by the following keys in :class:`~openturns.ResourceMap`:
    - *ResourceMap.SetAsBooletAsBool('GeneralLinearModelAlgorithm-UseAnalyticalAmplitudeEstimate', true)* to use the reduction associated to :math:`\sigma`. It has no effect if :math:`d>1` or if :math:`d=1` and :math:`\sigma` is not part of :math:`\vect{p}`
    - *ResourceMap.SetAsBool('GeneralLinearModelAlgorithm-UnbiasedVariance', true)* allows to use the *unbiased* estimate of :math:`sigma` where :math:`\dfrac{1}{N}` is replaced by :math:`\dfrac{1}{N-p}` in the optimality condition for :math:`\sigma`.

With huge samples, the `hierarchical matrix <http://en.wikipedia.org/wiki/Hierarchical_matrix>`_  implementation could be used if OpenTURNS had been compiled with `hmat-oss` support.

This implementation, which is based on a compressed representation of an approximated covariance matrix (and its Cholesky factor), has a better complexity both in terms of memory requirements and floating point operations. To use it, the `GeneralLinearModelAlgorithm-LinearAlgebra` resource map key should be instancied to `HMAT`. Default value of the key is `LAPACK`.

A known centered gaussian observation noise :math:`\epsilon_k` can be taken into account
with :func:`setNoise()`:

.. math:: \hat{\vect{y}}_k = \vect{y}_k + \epsilon_k, \epsilon_k \sim \mathcal{N}(0, \tau_k^2)

Examples
--------
Create the model :math:`\cM: \Rset \mapsto \Rset` and the samples:

>>> import openturns as ot
>>> f = ot.SymbolicFunction(['x'], ['x * sin(x)'])
>>> inputSample = ot.Sample([[1.0], [3.0], [5.0], [6.0], [7.0], [8.0]])
>>> outputSample = f(inputSample)

Create the algorithm:

>>> basis = ot.ConstantBasisFactory().build()
>>> covarianceModel = ot.SquaredExponential(1)
>>> algo = ot.GeneralLinearModelAlgorithm(inputSample, outputSample, covarianceModel, basis)
>>> algo.run()

Get the resulting meta model:

>>> result = algo.getResult()
>>> metamodel = result.getMetaModel()"

// ---------------------------------------------------------------------

%feature("docstring") OT::GeneralLinearModelAlgorithm::getInputTransformation
"Get the function normalizing the input.

Returns
-------
transformation : :class:`~openturns.Function`
    Function *T* that normalizes the input."

// ---------------------------------------------------------------------

%feature("docstring") OT::GeneralLinearModelAlgorithm::setInputTransformation
"Set the function normalizing the input.

Parameters
----------
transformation : :class:`~openturns.Function`
    Function that normalizes the input.
    The input dimension should be the same as input's sample dimension, output dimension
    should be output sample's dimension"

// ---------------------------------------------------------------------

%feature("docstring") OT::GeneralLinearModelAlgorithm::getResult
"Get the results of the metamodel computation.

Returns
-------
result : :class:`~openturns.GeneralLinearModelResult`
    Structure containing all the results obtained after computation
    and created by the method :py:meth:`run`.
"

//-----------------------------------------------------------------------

%feature("docstring") OT::GeneralLinearModelAlgorithm::getInputSample
"Accessor to the input sample.

Returns
-------
inputSample : :class:`~openturns.Sample`
    The input sample :math:`(\vect{x}_k)_{1 \leq k \leq N}`."

// ---------------------------------------------------------------------

%feature("docstring") OT::GeneralLinearModelAlgorithm::getOutputSample
"Accessor to the output sample.

Returns
-------
outputSample : :class:`~openturns.Sample`
    The output sample :math:`(\vect{y}_k)_{1 \leq k \leq N}` ."

// ---------------------------------------------------------------------

%feature("docstring") OT::GeneralLinearModelAlgorithm::getObjectiveFunction()
"Accessor to the log-likelihood function that writes as argument of the covariance's model parameters.

Returns
-------
logLikelihood : :class:`~openturns.Function`
    The log-likelihood function degined in :eq:`logLikelihood` as a function of :math:`(\vect{\theta}, \vect{\sigma})`.

Notes
-----
The log-likelihood function may be useful for some postprocessing: maximization using external optimizers for example.


Examples
--------
Create the model :math:`\cM: \Rset \mapsto \Rset` and the samples:

>>> import openturns as ot
>>> f = ot.SymbolicFunction(['x0'], ['x0 * sin(x0)'])
>>> inputSample = ot.Sample([[1.0], [3.0], [5.0], [6.0], [7.0], [8.0]])
>>> outputSample = f(inputSample)

Create the algorithm:

>>> basis = ot.ConstantBasisFactory().build()
>>> covarianceModel = ot.SquaredExponential(1)
>>> algo = ot.GeneralLinearModelAlgorithm(inputSample, outputSample, covarianceModel, basis)
>>> algo.run()

Get the log-likelihood function:

>>> likelihoodFunction = algo.getObjectiveFunction()
"


// ---------------------------------------------------------------------

%feature("docstring") OT::GeneralLinearModelAlgorithm::run
"Compute the response surface.

Notes
-----
It computes the response surface and creates a
:class:`~openturns.GeneralLinearModelResult` structure containing all the results."

// ---------------------------------------------------------------------

%feature("docstring") OT::GeneralLinearModelAlgorithm::getOptimizationAlgorithm
"Accessor to solver used to optimize the covariance model parameters.

Returns
-------
algorithm : :class:`~openturns.OptimizationAlgorithm`
    Solver used to optimize the covariance model parameters.
    Default optimizer is :class:`~openturns.TNC`"

// ---------------------------------------------------------------------

%feature("docstring") OT::GeneralLinearModelAlgorithm::setOptimizationAlgorithm
"Accessor to the solver used to optimize the covariance model parameters.

Parameters
----------
algorithm : :class:`~openturns.OptimizationAlgorithm`
    Solver used to optimize the covariance model parameters."

// ---------------------------------------------------------------------

%feature("docstring") OT::GeneralLinearModelAlgorithm::setOptimizeParameters
"Accessor to the covariance model parameters optimization flag.

Parameters
----------
optimizeParameters : bool
    Whether to optimize the covariance model parameters."

// ---------------------------------------------------------------------

%feature("docstring") OT::GeneralLinearModelAlgorithm::getOptimizeParameters
"Accessor to the covariance model parameters optimization flag.

Returns
-------
optimizeParameters : bool
    Whether to optimize the covariance model parameters."

// ---------------------------------------------------------------------

%feature("docstring") OT::GeneralLinearModelAlgorithm::setOptimizationBounds
"Optimization bounds accessor.

Parameters
----------
bounds : :class:`~openturns.Interval`
    Bounds for covariance model parameter optimization."

// ---------------------------------------------------------------------

%feature("docstring") OT::GeneralLinearModelAlgorithm::getOptimizationBounds
"Optimization bounds accessor.

Returns
-------
bounds : :class:`~openturns.Interval`
    Bounds for covariance model parameter optimization."

// ---------------------------------------------------------------------

%feature("docstring") OT::GeneralLinearModelAlgorithm::setNoise
"Observation noise variance accessor.

Parameters
----------
noise : sequence of positive float
    The noise variance :math:`\tau_k^2` of each output value."

// ---------------------------------------------------------------------

%feature("docstring") OT::GeneralLinearModelAlgorithm::getNoise
"Observation noise variance accessor.

Parameters
----------
noise : sequence of positive float
    The noise variance :math:`\tau_k^2` of each output value."

