\chapter{Cheat sheet}
\label{chap:cheatsheet}

This chapter explains how to perform some common---and some not so
common---tasks in gretl's scripting language, hansl.  Some but not
all of the techniques listed here are also available through the
graphical interface.  Although the graphical interface may be more
intuitive and less intimidating at first, we encourage users to take
advantage of the power of gretl's scripting language as soon as
they feel comfortable with the program.

\section{Dataset handling}

\subsection{``Weird'' periodicities}

\emph{Problem:} You have data sampled each 3 minutes from 9am onwards;
you'll probably want to specify the hour as 20 periods.

\emph{Solution:}
\begin{code}
setobs 20 9:1 --special
\end{code}

\emph{Comment:} Now functions like \cmd{sdiff()} (``seasonal''
difference) or estimation methods like seasonal ARIMA will work as
expected.

\subsection{Generating a panel dataset of given dimensions}

\emph{Problem:} You want to generate via \cmd{nulldata} a panel
dataset and specify in advance the number of units and the time length
of your series via two scalar variables. 

\emph{Solution:}
\begin{code}
scalar n_units = 100
scalar T = 12
scalar NT = T * n_units

nulldata NT --preserve
setobs T 1:1 --stacked-time-series
\end{code}

\emph{Comment:} The essential ingredient that we use here is the
\option{preserve} option: it protects existing scalars (and matrices,
for that matter) from being trashed by \cmd{nulldata}, thus making it
possible to use the scalar $T$ in the \cmd{setobs} command.

\subsection{Help, my data are backwards!}

\emph{Problem:} Gretl expects time series data to be in chronological
order (most recent observation last), but you have imported
third-party data that are in reverse order (most recent first).

\emph{Solution:}
\begin{code}
setobs 1 1 --cross-section
series sortkey = -obs
dataset sortby sortkey
setobs 1 1950 --time-series
\end{code}

\emph{Comment:} The first line is required only if the data currently
have a time series interpretation: it removes that interpretation,
because (for fairly obvious reasons) the \cmd{dataset sortby}
operation is not allowed for time series data.  The following two
lines reverse the data, using the negative of the built-in index
variable \texttt{obs}.  The last line is just illustrative: it
establishes the data as annual time series, starting in 1950.

If you have a dataset that is mostly the right way round, but a
particular variable is wrong, you can reverse that variable as
follows:
\begin{code}
x = sortby(-obs, x)
\end{code}


\subsection{Dropping missing observations selectively}

\emph{Problem:} You have a dataset with many variables and want to
restrict the sample to those observations for which there are no
missing observations for the variables \texttt{x1}, \texttt{x2} and
\texttt{x3}.

\begin{samepage}
\emph{Solution:}
\begin{code}
list X = x1 x2 x3
smpl --no-missing X
\end{code}
\end{samepage}

\emph{Comment:} You can now save the file via a \cmd{store} command
to preserve a subsampled version of the dataset. Alternative solutions
based on the \cmd{ok} function, such as
\begin{code}
list X = x1 x2 x3
series sel = ok(X)
smpl sel --restrict
\end{code}
are perhaps less obvious, but more flexible. Pick your poison.

\subsection{``By'' operations}

\emph{Problem:} You have a discrete variable \texttt{d} and you want
to run some commands (for example, estimate a model) by splitting the
sample according to the values of \texttt{d}.

\emph{Solution:}
\begin{code}
matrix vd = values(d)
m = rows(vd)
loop i=1..m
  scalar sel = vd[i]
  smpl d==sel --restrict --replace
  ols y const x
endloop
smpl --full
\end{code}

\emph{Comment:} The main ingredient here is a loop.  You can have
gretl perform as many instructions as you want for each value of
\texttt{d}, as long as they are allowed inside a loop. Note, however,
that if all you want is descriptive statistics, the \cmd{summary}
command does have a \option{by} option.

\subsection{Adding a time series to a panel}

\emph{Problem:} You have a panel dataset (comprising observations of
$n$ indidivuals in each of $T$ periods) and you want to add a variable
which is available in straight time-series form.  For example, you
want to add annual CPI data to a panel in order to deflate nominal
income figures.

In gretl a panel is represented in stacked time-series format, so in
effect the task is to create a new variable which holds $n$ stacked
copies of the original time series.  Let's say the panel comprises 500
individuals observed in the years 1990, 1995 and 2000 ($n=500$,
$T=3$), and we have these CPI data in the ASCII file \texttt{cpi.txt}:

\begin{code}
date cpi
1990 130.658
1995 152.383
2000 172.192
\end{code}

What we need is for the CPI variable in the panel to repeat these
three values 500 times.

\emph{Solution:} Simple!  With the panel dataset open in gretl,
\begin{code}
append cpi.txt
\end{code}

\emph{Comment:} If the length of the time series is the same as the
length of the time dimension in the panel (3 in this example), gretl
will perform the stacking automatically.  Rather than using the
\cmd{append} command you could use the ``Append data'' item under
the \textsf{File} menu in the GUI program.  For this to work, your
main dataset must be recognized as a panel.  This can be arranged via
the \cmd{setobs} command or the ``Dataset structure'' item under
the \textsf{Data} menu.


\section{Creating/modifying variables}

\subsection{Generating a dummy variable for a specific observation}

\emph{Problem:} Generate $d_t = 0$ for all observation but one, for
which $d_t = 1$.

\emph{Solution:}
\begin{code}
  series d = (t=="1984:2")
\end{code}

\emph{Comment:} The internal variable \texttt{t} is used to refer to
observations in string form, so if you have a cross-section sample you
may just use \texttt{d = (t=="123")}. If the dataset has observation
labels you can use the corresponding label. For example, if you open the
dataset \texttt{mrw.gdt}, supplied with gretl among the
examples, a dummy variable for Italy could be generated via
\begin{code}
  series DIta = (t=="Italy")
\end{code}

Note that this method does not require scripting at all. In fact, you
might as well use the GUI Menu ``Add/Define new variable'' for the
same purpose, with the same syntax.

\subsection{Generating a discrete variable out of a set of dummies}

\emph{Problem:} The \cmd{dummify} function (also available as a
command) generates a set of mutually exclusive dummies from a discrete
variable. The reverse functionality, however, seems to be absent.

\emph{Solution:}
\begin{code}
series x = lincomb(D, seq(1, nelem(D)))
\end{code}

\emph{Comment:} Suppose you have a list \texttt{D} of mutually
exclusive dummies, that is a full set of 0/1 variables coding for the
value of some characteristic, such that the sum of the values of the
elements of \texttt{D} is 1 at each observation. This is, by the way,
exactly what the \cmd{dummify} command produces.  The reverse job of
\cmd{dummify} can be performed neatly by using the \cmd{lincomb}
function.

The code above multiplies the first dummy variable in the list
\texttt{D} by 1, the second one by 2, and so on. Hence, the return
value is a series whose value is $i$ if and only if the $i$-th member
of \texttt{D} has value 1.

If you want your coding to start from 0 instead of 1, you'll have to
modify the code snippet above into
\begin{code}
series x = lincomb(D, seq(0, nelem(D)-1))
\end{code}

\subsection{Generating an ARMA(1,1)}

\emph{Problem:} Generate $y_t = 0.9 y_{t-1} + \varepsilon_t - 0.5
\varepsilon_{t-1}$, with $\varepsilon_t \sim N\!I\!I\!D(0,1)$.

\emph{Recommended solution:}
\begin{code}
alpha = 0.9
theta = -0.5
series y = filter(normal(), {1, theta}, alpha)
\end{code}

\emph{``Bread and butter'' solution:}
\begin{code}
alpha = 0.9
theta = -0.5
series e = normal()
series y = 0
series y = alpha * y(-1) + e + theta * e(-1)
\end{code}

\emph{Comment:} The \cmd{filter} function is specifically designed for
this purpose so in most cases you'll want to take advantage of its
speed and flexibility. That said, in some cases you may want to
generate the series in a manner which is more transparent (maybe for
teaching purposes). 

In the second solution, the statement \cmd{series y = 0} is necessary
because the next statement evaluates \texttt{y} recursively, so
\texttt{y[1]} must be set.  Note that you must use the keyword
\texttt{series} here instead of writing \texttt{genr y = 0} or simply
\texttt{y = 0}, to ensure that \texttt{y} is a series and
not a scalar.

\subsection{Recoding a variable}

\emph{Problem:} You want to perform a 1-to-1 recode on a variable. For
example, consider tennis points: you may have a variable $x$ holding
values 1 to 3 and you want to recode it to 15, 30, 40. 

\emph{Solution 1:}
\begin{code}
series x = replace(x, 1, 15)
series x = replace(x, 2, 30)
series x = replace(x, 3, 40)
\end{code}

\emph{Solution 2:}
\begin{code}
matrix tennis = {15, 30, 40}
series x = replace(x, seq(1,3), tennis)
\end{code}

\emph{Comment:} There are many equivalent ways to achieve the same
effect, but for simple cases such as this, the \texttt{replace}
function is simple and transparent. If you don't mind using matrices,
scripts using \texttt{replace} can also be remarkably compact.

\subsection{Recoding a variable by classes}

\emph{Problem:} You want to recode a variable by classes. For example,
you have the age of a sample of individuals ($x_i$) and you need to
compute age classes ($y_i$) as
\begin{eqnarray*}
  y_i = 1 & \mathrm{for} & x_i < 18 \\
  y_i = 2 & \mathrm{for} & 18 \le x_i < 65 \\
  y_i = 3 & \mathrm{for} & x_i \ge 65
\end{eqnarray*}

\emph{Solution:}
\begin{code}
series y = 1 + (x >= 18) + (x >= 65)
\end{code}

\emph{Comment:} True and false expressions are evaluated as 1 and 0
respectively, so they can be manipulated algebraically as any other
number. The same result could also be achieved by using the
conditional assignment operator (see below), but in most cases it
would probably lead to more convoluted constructs.

\subsection{Conditional assignment}

\emph{Problem:} Generate $y_t$ via the following rule:
\[
  y_t = \left\{ 
    \begin{array}{ll} 
      x_t & \mathrm{for} \quad d_t > a \\ 
      z_t & \mathrm{for} \quad d_t \le a 
    \end{array}
    \right. 
\]

\emph{Solution:}
\begin{code}
series y = (d > a) ? x : z
\end{code}

\emph{Comment:} There are several alternatives to the one presented
above. One is a brute force solution using loops. Another one, more
efficient but still suboptimal, would be 
\begin{code}
series y = (d>a)*x + (d<=a)*z  
\end{code}
However, the ternary conditional assignment operator is not only the
most numerically efficient way to accomplish what we want, it is also
remarkably transparent to read when one gets used to it. Some readers
may find it helpful to note that the conditional assignment operator
works exactly the same way as the \texttt{=IF()} function in
spreadsheets.

\subsection{Generating a time index for panel datasets}

\emph{Problem:} gretl has a \texttt{\$unit} accessor, but not
the equivalent for time. What should I use?

\emph{Solution:}
\begin{code}
series x = time
\end{code}

\emph{Comment:} The special construct \cmd{genr time} and its variants
are aware of whether a dataset is a panel.


\subsection{Generating the ``hat'' values after an OLS regression}

\emph{Problem:} I've just run an OLS regression, and now I need the
so-called the leverage values (also known as the ``hat'' values). I
know you can access residuals and fitted values through ``dollar''
accessors, but nothing like that seems to be available for ``hat''
values.

\emph{Solution:}
``Hat'' values are can be thought of as the diagonal of the projection
matrix $P_X$, or more explicitly as
\[
h_i = \mathbf{x}_i' (X'X)^{-1} \mathbf{x}_i 
\]
where $X$ is the matrix of regressors and $\mathbf{x}_i'$ is its
$i$-th row.

The reader is invited to study the code below, which offers four
different solutions to the problem:
 
\begin{code}
open data4-1.gdt --quiet
list X = const sqft bedrms baths
ols price X

# method 1
leverage --save --quiet
series h1 = lever

# these are necessary for what comes next
matrix mX  = {X}
matrix iXX = invpd(mX'mX)

# method 2
series h2 = diag(qform(mX, iXX))
# method 3
series h3 = sumr(mX .* (mX*iXX))
# method 4
series h4 = NA
loop i=1..$nobs --quiet
    matrix x = mX[i,]'
    h4[i] = x'iXX*x
endloop

# verify
print h1 h2 h3 h4 --byobs
\end{code}
%$

\emph{Comment:} Solution 1 is the preferable one: it relies on the
built-in \cmd{leverage} command, which computes the requested series
quite efficiently, taking care of missing values, possible
restrictions to the sample, etcetera.

However, three more are shown for didactical purposes, mainly to show
the user how to manipulate matrices. Solution 2 first constructs the
$P_X$ matrix explicitly, via the \cmd{qform} function, and then takes
its diagonal; this is definitely \emph{not} recommended (despite its
compactness), since you generate a much bigger matrix than you
actually need and waste a lot of memory and CPU cycles in the
process. It doesn't matter very much in the present case, since the
sample size is very small, but with a big dataset this could be a very
bad idea.

Solution 3 is more clever, and relies on the fact that, if you
define $Z = X\cdot(X'X)^{-1}$, then $h_i$ could also
be written as
\[
 h_i = \mathbf{x}_i' \mathbf{z}_i = \sum_{i=1}^k x_{ik} z_{i_k} 
\]
which is in turn equivalent to the sum of the elements of the $i$-th
row of $X \odot Z$, where $\odot$ is the element-by-element
product. In this case, your clever usage of matrix algebra would
produce a solution computationally much superior to solution 2.

Solution 4 is the most old-fashioned one, and employs an indexed
loop. While this wastes practically no memory and employs no more CPU
cycles in algebraic operations than strictly necessary, it imposes a
much greater burden on the hansl interpreter, since handling a loop is
conceptually more complex than a single operation. In practice, you'll
find that for any realistically-sized problem, solution 4 is much
slower that solution 3.

\section{Neat tricks}
\label{sec:cheat-neat}

\subsection{Interaction dummies}

\emph{Problem:} You want to estimate the model $y_i = \mathbf{x}_i
\beta_1 + \mathbf{z}_i \beta_2 + d_i \beta_3 + (d_i \cdot \mathbf{z}_i)
\beta_4 + \varepsilon_t$, where $d_i$ is a dummy variable while
$\mathbf{x}_i$ and $\mathbf{z}_i$ are vectors of explanatory
variables.

\emph{Solution:} As of version 1.9.12, gretl provides the
\verb|^| operator to make this operation easy. See section
\ref{sec:transform-lists} for details (especially example script
\ref{ex:interact_list}). But back in my day, we used loops to do that!
Here's how:

\begin{code}
list X = x1 x2 x3
list Z = z1 z2
list dZ = null
loop foreach i Z
  series d$i = d * $i
  list dZ = dZ d$i
endloop 

ols y X Z d dZ
\end{code} 
%$

\emph{Comment:} It's amazing what string substitution can do for
you, isn't it?

\subsection{Realized volatility}

\emph{Problem:} Given data by the minute, you want to compute the
``realized volatility'' for the hour as $RV_t = \frac{1}{60}
\sum_{\tau=1}^{60} y_{t:\tau}^2$. Imagine your sample starts at time 1:1.

\emph{Solution:}
\begin{code}
smpl --full
genr time
series minute = int(time/60) + 1
series second = time % 60
setobs minute second --panel
series rv = psd(y)^2
setobs 1 1
smpl second==1 --restrict
store foo rv
\end{code}

\emph{Comment:} Here we trick gretl into thinking that our
dataset is a panel dataset, where the minutes are the ``units'' and
the seconds are the ``time''; this way, we can take advantage of the
special function \cmd{psd()}, panel standard deviation.  Then we
simply drop all observations but one per minute and save the resulting
data (\texttt{store foo rv} translates as ``store in the gretl
datafile \texttt{foo.gdt} the series \texttt{rv}'').

\subsection{Looping over two paired lists}

\emph{Problem:} Suppose you have two lists with the same number of
elements, and you want to apply some command to corresponding elements
over a loop.

\emph{Solution:}
\begin{code}
list L1 = a b c
list L2 = x y z

k1 = 1
loop foreach i L1 --quiet
    k2 = 1
    loop foreach j L2 --quiet
        if k1 == k2
            ols $i 0 $j
        endif
        k2++
    endloop
    k1++
endloop
\end{code}

\emph{Comment:} The simplest way to achieve the result is to loop over
all possible combinations and filter out the unneeded ones via an
\texttt{if} condition, as above. That said, in some cases variable
names can help. For example, if
\begin{code}
  list Lx = x1 x2 x3
  list Ly = y1 y2 y3
\end{code}
then we could just loop over the integers---quite intuitive and certainly 
more elegant:
\begin{code}
  loop i=1..3
    ols y$i const x$i
  endloop
\end{code}

\subsection{Dropping collinear variables}

\emph{Problem:} I'm writing my own estimator and I'd like to have a
function that automatically drops collinear variables from a list.

\emph{Solution:}
The neatest way to achieve this is to use \emph{Gram-Schmidt
  orthogonalization}, which is performed by the matrix function
\cmd{qrdecomp()}. The function below defines a user-level function
that (i) converts the list to a matrix (ii) spots collinear columns
and (iii) removes them if any.

\begin{code}
function list autodrop(list X, scalar criterion[1.0e-12])
    list drop = null
    matrix R
    qrdecomp({X}, &R)

    scalar ndrop = 0
    loop i=1..nelem(X) --quiet
        if criterion > abs(R[i,i])
            drop += X[i]
            ndrop++
        endif
    endloop

    if ndrop > 0 
        printf "%d collinear variables dropped (%s)\n", \
          ndrop, varname(drop)    
        return X - drop
    else 
        return X
    endif
end function
\end{code}

\emph{Comment:} The function also accepts a second optional argument
which sets a criterion for deciding how strict the decision rule must
be. Usage example:

\begin{code}
nulldata 20

x1 = normal()
x2 = normal()
x3 = x1 + x2
x4 = x1 + 1

list A = const x1 x2 x3 x4
list C = autodrop(A)
\end{code}
Running the code above will create a list \texttt{C} containing just
\texttt{const}, \texttt{x1} and \texttt{x2}.

\subsection{Comparing two lists}

\emph{Problem:} How can I tell if two lists contain the same variables
(not necessarily in the same order)?

\emph{Solution:} Under many respects, lists are like sets, so it makes
sense to use the so-called ``symmetric difference'' operator; this, in
turn can be defined as
\[
A\,\triangle\,B = (A \backslash B) \cup (B \backslash A) .
\]

In practice: we first check if there are series in $A$ but not in $B$,
then we perform the reverse check. If the union of the two results is
an empty set, then the lists contain the same variables. The hansl
syntax for this would be something like

\begin{code}
scalar NotTheSame = nelem((A-B) || (B-A)) > 0
\end{code}

%%% Local Variables: 
%%% mode: latex
%%% TeX-master: "gretl-guide"
%%% End: 
