.. _adding_metadata:

====================================================
Adding sample and observation metadata to biom files
====================================================

Frequently you'll have an existing BIOM file and want to add sample and/or observation metadata to it. For samples, metadata is frequently environmental or technical details about your samples: the subject that a sample was collected from, the pH of the sample, the PCR primers used to amplify DNA from the samples, etc. For observations, metadata is frequently a categorization of the observation: the taxonomy of an OTU, or the EC hierarchy of a gene. You can use the ``biom add-metadata`` command to add this information to an existing BIOM file.

To get help with ``add-metadata`` you can call::

	biom add-metadata -h

This command takes a BIOM file, and corresponding sample and/or observation mapping files. The following examples are used in the commands below. You can find these files in the ``biom-format/examples`` directory.

Your BIOM file might look like the following::

    {
        "id":null,
        "format": "1.0.0",
        "format_url": "http://biom-format.org",
        "type": "OTU table",
        "generated_by": "some software package",
        "date": "2011-12-19T19:00:00",
        "rows":[
                {"id":"GG_OTU_1", "metadata":null},
                {"id":"GG_OTU_2", "metadata":null},
                {"id":"GG_OTU_3", "metadata":null},
                {"id":"GG_OTU_4", "metadata":null},
                {"id":"GG_OTU_5", "metadata":null}
            ],  
        "columns": [
                {"id":"Sample1", "metadata":null},
                {"id":"Sample2", "metadata":null},
                {"id":"Sample3", "metadata":null},
                {"id":"Sample4", "metadata":null},
                {"id":"Sample5", "metadata":null},
                {"id":"Sample6", "metadata":null}
            ],
        "matrix_type": "sparse",
        "matrix_element_type": "int",
        "shape": [5, 6], 
        "data":[[0,2,1],
                [1,0,5],
                [1,1,1],
                [1,3,2],
                [1,4,3],
                [1,5,1],
                [2,2,1],
                [2,3,4],
                [2,5,2],
                [3,0,2],
                [3,1,1],
                [3,2,1],
                [3,5,1],
                [4,1,1],
                [4,2,1]
               ]
    }

A sample metadata mapping file could then look like the following. Notice that there is an extra sample in here with respect to the above BIOM table. Any samples in the mapping file that are not in the BIOM file are ignored.
::

	#SampleID	BarcodeSequence	DOB
	# Some optional
	# comment lines...
	Sample1	AGCACGAGCCTA	20060805
	Sample2	AACTCGTCGATG	20060216
	Sample3	ACAGACCACTCA	20060109
	Sample4	ACCAGCGACTAG	20070530
	Sample5	AGCAGCACTTGT	20070101
	Sample6	AGCAGCACAACT	20070716

An observation metadata mapping file might look like the following. Notice that there is an extra observation in here with respect to the above BIOM table. Any observations in the mapping file that are not in the BIOM file are ignored.
::

	#OTUID	taxonomy	confidence
	# Some optional
	# comment lines
	GG_OTU_0	Root;k__Bacteria;p__Firmicutes;c__Clostridia;o__Clostridiales;f__	0.980
	GG_OTU_1	Root;k__Bacteria;p__Firmicutes;c__Clostridia;o__Clostridiales;f__Lachnospiraceae	0.665
	GG_OTU_2	Root;k__Bacteria	0.980
	GG_OTU_3	Root;k__Bacteria;p__Firmicutes;c__Clostridia;o__Clostridiales;f__Lachnospiraceae	1.000
	GG_OTU_4	Root;k__Bacteria;p__Firmicutes;c__Clostridia;o__Clostridiales;f__Lachnospiraceae	0.842
	GG_OTU_5	Root;k__Bacteria;p__Firmicutes;c__Clostridia;o__Clostridiales;f__Lachnospiraceae	1.000


Adding metadata
===============

To add sample metadata to a BIOM file, you can run the following::

	biom add-metadata -i min_sparse_otu_table.biom -o table.w_smd.biom --sample-metadata-fp sam_md.txt

To add observation metadata to a BIOM file, you can run the following::

	biom add-metadata -i min_sparse_otu_table.biom -o table.w_omd.biom --observation-metadata-fp obs_md.txt

You can also combine these in a single command to add both observation and sample metadata::

	biom add-metadata -i min_sparse_otu_table.biom -o table.w_md.biom --observation-metadata-fp obs_md.txt --sample-metadata-fp sam_md.txt

In the last case, the resulting BIOM file will look like the following::

	{
	    "columns": [
	        {
	            "id": "Sample1", 
	            "metadata": {
	                "BarcodeSequence": "AGCACGAGCCTA", 
	                "DOB": "20060805"
	            }
	        }, 
	        {
	            "id": "Sample2", 
	            "metadata": {
	                "BarcodeSequence": "AACTCGTCGATG", 
	                "DOB": "20060216"
	            }
	        }, 
	        {
	            "id": "Sample3", 
	            "metadata": {
	                "BarcodeSequence": "ACAGACCACTCA", 
	                "DOB": "20060109"
	            }
	        }, 
	        {
	            "id": "Sample4", 
	            "metadata": {
	                "BarcodeSequence": "ACCAGCGACTAG", 
	                "DOB": "20070530"
	            }
	        }, 
	        {
	            "id": "Sample5", 
	            "metadata": {
	                "BarcodeSequence": "AGCAGCACTTGT", 
	                "DOB": "20070101"
	            }
	        }, 
	        {
	            "id": "Sample6", 
	            "metadata": {
	                "BarcodeSequence": "AGCAGCACAACT", 
	                "DOB": "20070716"
	            }
	        }
	    ], 
	    "data": [
	        [0, 2, 1.0], 
	        [1, 0, 5.0], 
	        [1, 1, 1.0], 
	        [1, 3, 2.0], 
	        [1, 4, 3.0], 
	        [1, 5, 1.0], 
	        [2, 2, 1.0], 
	        [2, 3, 4.0], 
	        [2, 5, 2.0], 
	        [3, 0, 2.0], 
	        [3, 1, 1.0], 
	        [3, 2, 1.0], 
	        [3, 5, 1.0], 
	        [4, 1, 1.0], 
	        [4, 2, 1.0]
	    ], 
	    "date": "2012-12-11T07:36:15.467843", 
	    "format": "Biological Observation Matrix 1.0.0", 
	    "format_url": "http://biom-format.org", 
	    "generated_by": "some software package", 
	    "id": null, 
	    "matrix_element_type": "float", 
	    "matrix_type": "sparse", 
	    "rows": [
	        {
	            "id": "GG_OTU_1", 
	            "metadata": {
	                "confidence": "0.665", 
	                "taxonomy": "Root;k__Bacteria;p__Firmicutes;c__Clostridia;o__Clostridiales;f__Lachnospiraceae"
	            }
	        }, 
	        {
	            "id": "GG_OTU_2", 
	            "metadata": {
	                "confidence": "0.980", 
	                "taxonomy": "Root;k__Bacteria"
	            }
	        }, 
	        {
	            "id": "GG_OTU_3", 
	            "metadata": {
	                "confidence": "1.000", 
	                "taxonomy": "Root;k__Bacteria;p__Firmicutes;c__Clostridia;o__Clostridiales;f__Lachnospiraceae"
	            }
	        }, 
	        {
	            "id": "GG_OTU_4", 
	            "metadata": {
	                "confidence": "0.842", 
	                "taxonomy": "Root;k__Bacteria;p__Firmicutes;c__Clostridia;o__Clostridiales;f__Lachnospiraceae"
	            }
	        }, 
	        {
	            "id": "GG_OTU_5", 
	            "metadata": {
	                "confidence": "1.000", 
	                "taxonomy": "Root;k__Bacteria;p__Firmicutes;c__Clostridia;o__Clostridiales;f__Lachnospiraceae"
	            }
	        }
	    ], 
	    "shape": [5, 6], 
	    "type": "OTU table"
	}


Processing metadata while adding
================================

There are some additional parameters you can pass to this command for more complex processing. 

You can tell the command to process certain metadata column values as integers (``--int-fields``), floating point (i.e., decimal or real) numbers (``--float-fields``), or as hierarchical semicolon-delimited data (``--sc-separated``).

::

	biom add-metadata -i min_sparse_otu_table.biom -o table.w_md.biom --observation-metadata-fp obs_md.txt --sample-metadata-fp sam_md.txt --int-fields DOB --sc-separated taxonomy --float-fields confidence

Here your resulting BIOM file will look like the following, where ``DOB`` values are now integers (compare to the above: they're not quoted now), ``confidence`` values are now floating point numbers (again, not quoted now), and ``taxonomy`` values are now lists where each entry is a taxonomy level, opposed to above where they appear as a single semi-colon-separated string.
::

	{
	    "columns": [
	        {
	            "id": "Sample1", 
	            "metadata": {
	                "BarcodeSequence": "AGCACGAGCCTA", 
	                "DOB": 20060805
	            }
	        }, 
	        {
	            "id": "Sample2", 
	            "metadata": {
	                "BarcodeSequence": "AACTCGTCGATG", 
	                "DOB": 20060216
	            }
	        }, 
	        {
	            "id": "Sample3", 
	            "metadata": {
	                "BarcodeSequence": "ACAGACCACTCA", 
	                "DOB": 20060109
	            }
	        }, 
	        {
	            "id": "Sample4", 
	            "metadata": {
	                "BarcodeSequence": "ACCAGCGACTAG", 
	                "DOB": 20070530
	            }
	        }, 
	        {
	            "id": "Sample5", 
	            "metadata": {
	                "BarcodeSequence": "AGCAGCACTTGT", 
	                "DOB": 20070101
	            }
	        }, 
	        {
	            "id": "Sample6", 
	            "metadata": {
	                "BarcodeSequence": "AGCAGCACAACT", 
	                "DOB": 20070716
	            }
	        }
	    ], 
	    "data": [
	        [0, 2, 1.0], 
	        [1, 0, 5.0], 
	        [1, 1, 1.0], 
	        [1, 3, 2.0], 
	        [1, 4, 3.0], 
	        [1, 5, 1.0], 
	        [2, 2, 1.0], 
	        [2, 3, 4.0], 
	        [2, 5, 2.0], 
	        [3, 0, 2.0], 
	        [3, 1, 1.0], 
	        [3, 2, 1.0], 
	        [3, 5, 1.0], 
	        [4, 1, 1.0], 
	        [4, 2, 1.0]
	    ], 
	    "date": "2012-12-11T07:30:29.870689", 
	    "format": "Biological Observation Matrix 1.0.0", 
	    "format_url": "http://biom-format.org", 
	    "generated_by": "some software package", 
	    "id": null, 
	    "matrix_element_type": "float", 
	    "matrix_type": "sparse", 
	    "rows": [
	        {
	            "id": "GG_OTU_1", 
	            "metadata": {
	                "confidence": 0.665, 
	                "taxonomy": ["Root", "k__Bacteria", "p__Firmicutes", "c__Clostridia", "o__Clostridiales", "f__Lachnospiraceae"]
	            }
	        }, 
	        {
	            "id": "GG_OTU_2", 
	            "metadata": {
	                "confidence": 0.98, 
	                "taxonomy": ["Root", "k__Bacteria"]
	            }
	        }, 
	        {
	            "id": "GG_OTU_3", 
	            "metadata": {
	                "confidence": 1.0, 
	                "taxonomy": ["Root", "k__Bacteria", "p__Firmicutes", "c__Clostridia", "o__Clostridiales", "f__Lachnospiraceae"]
	            }
	        }, 
	        {
	            "id": "GG_OTU_4", 
	            "metadata": {
	                "confidence": 0.842, 
	                "taxonomy": ["Root", "k__Bacteria", "p__Firmicutes", "c__Clostridia", "o__Clostridiales", "f__Lachnospiraceae"]
	            }
	        }, 
	        {
	            "id": "GG_OTU_5", 
	            "metadata": {
	                "confidence": 1.0, 
	                "taxonomy": ["Root", "k__Bacteria", "p__Firmicutes", "c__Clostridia", "o__Clostridiales", "f__Lachnospiraceae"]
	            }
	        }
	    ], 
	    "shape": [5, 6], 
	    "type": "OTU table"
	}

If you have multiple fields that you'd like processed in one of these ways, you can pass a comma-separated list of field names (e.g., ``--float-fields confidence,pH``).

Renaming (or naming) metadata columns while adding
==================================================

You can also override the names of the metadata fields provided in the mapping files with the ``--observation-header`` and ``--sample-header`` parameters. This is useful if you want to rename metadata columns, or if metadata column headers aren't present in your metadata mapping file. If you pass either of these parameters, you must name all columns in order. If there are more columns in the metadata mapping file then there are headers, extra columns will be ignored (so this is also a useful way to select only the first n columns from your mapping file). For example, if you want to rename the ``DOB`` column in the sample metadata mapping you could do the following::
	
	biom add-metadata -i min_sparse_otu_table.biom -o table.w_smd.biom --sample-metadata-fp sam_md.txt --sample-header SampleID,BarcodeSequence,DateOfBirth

If you have a mapping file without headers such as the following::

	GG_OTU_0	Root;k__Bacteria;p__Firmicutes;c__Clostridia;o__Clostridiales;f__	0.980
	GG_OTU_1	Root;k__Bacteria;p__Firmicutes;c__Clostridia;o__Clostridiales;f__Lachnospiraceae	0.665
	GG_OTU_2	Root;k__Bacteria	0.980
	GG_OTU_3	Root;k__Bacteria;p__Firmicutes;c__Clostridia;o__Clostridiales;f__Lachnospiraceae	1.000
	GG_OTU_4	Root;k__Bacteria;p__Firmicutes;c__Clostridia;o__Clostridiales;f__Lachnospiraceae	0.842
	GG_OTU_5	Root;k__Bacteria;p__Firmicutes;c__Clostridia;o__Clostridiales;f__Lachnospiraceae	1.000

you could name these while adding them as follows::

	biom add-metadata -i min_sparse_otu_table.biom -o table.w_omd.biom --observation-metadata-fp obs_md.txt --observation-header OTUID,taxonomy,confidence

As a variation on the last command, if you only want to include the ``taxonomy`` column and exclude the ``confidence`` column, you could run::

	biom add-metadata -i min_sparse_otu_table.biom -o table.w_omd.biom --observation-metadata-fp obs_md.txt --observation-header OTUID,taxonomy --sc-separated taxonomy

