---
title: "Tracking Artifacts"
sub_link: "tracking/artifacts"
meta_title: "Add artifact tracking and log metadata to the lineage table - Tracking - Experimentation"
meta_description: "Tracking artifacts and log metadata lineage related to files and assets."
visibility: public
status: published
tags:
  - client
  - api
  - polyaxon
  - python
  - tracking
  - reference
  - sdk
sidebar: "experimentation"
---

Polyaxon provides two methods for tracking assets and artifacts that you generate in your jobs:

 * Versioned assets.
 * Reference-only lineage metadata.
 
## Overview

For each run, Polyaxon creates an artifacts folder with a predefined structure to organize your assets and outputs.
 * outputs: This is where the user can put custom artifacts that are not generated by Polyaxon library, additionally both `log_artifact(...)` and `log_model(...)` will move the file/dir under `outputs` if no `step` is provided.
 * assets: This folder is populated automatically when a `log_...` method is called, an event file and an asset gets created.
 * events: This folder is populated automatically when a `log_...` method is called.
 * plxlogs: This folder is populated automatically if archiving logs is enabled.
 * resources: This folder is populated automatically if resources tracking is enabled.

Users might also save content in other subpaths as they wish, for instance, uploading data to `uploads` or `code`.

> **Note**: during the runtime, users should only save their custom artifacts under `outputs` to leverage automatic artifact management,
> we suggest using [get_outputs_path](/docs/experimentation/tracking/client/#get_outputs_path) for getting and ensuring a new outputs subpath.

For each run, users can get the artifacts_path and outputs_path using:

 * `tracking.get_artifacts_path()`
 * `tracking.get_outputs_path()`
 
They can also get a relative path to these root paths:

 * `tracking.get_artifacts_path("code/file.py")`
 * `tracking.get_outputs_path("model/model.h5")`
 * `tracking.get_outputs_path("sub-folder", is_dir=True)`

Polyaxon also exposes 3 types of logging methods:

 * Reference logging: these are the methods ending with `_ref`, they generally only log the lineage reference, the user has to save the artifact manually.
 * Versioned assets logging: These are the methods that save both the assets and the corresponding event file.
 * Generic methods: `log_artifact` and `log_model`, these two methods can save versioned assets (default behavior) or move the asset under the `outputs` folder and save the lineage is `versioned=False`.

## Reference logging

Reference logging is useful when the user:

 * needs to have more control where the asset must be saved.
 * does not need to log multiple versions of the asset during the run, i.e. the asset is only saved once.
 
In reference logging the user is responsible for saving the artifacts, you can use a relative path to the run's root artifacts or a relative path to the run's outputs path:

 * `tracking.get_outputs_path("model/model.h5")`
 
After saving the artifact under that path, you can decide if you want to create a new entry in the lineage table:

```python
asset_path = tracking.get_outputs_path("custom_artifacts/filename.ext")
custom_logic_to_save_the_file(asset_path)

# This is optional, it tracks lineage only
tracking.log_artifact_ref(path=asset_path, ...)
```

Sometimes you will need to save an artifact on different backend and not on the artifact store, you can still use the ref to log the lineage:

```python
asset_path = "{}/file.ext".format(S3_URI)
custom_logic_to_save_in_s3(asset_path)

# This is optional, it tracks lineage only
tracking.log_artifact_ref(path=asset_path, name="myfile", summary={"extra_key": "extra_value"}, ...)
```

In this case the file was not saved to the default artifacts store, but it was saved to a custom S3 bucket, and we added a new entry in the lineage table with that information.

Saving a model reference

```python
from polyaxon import tracking
...
tracking.init()
...
model_dir = tracking.get_outputs_path("model", is_dir=True)
classifier = tf.estimator.LinearClassifier(
    model_dir=model_dir,
    feature_columns=[...],
    n_classes=2
)
tracking.log_model_ref(model_dir, framework="tensorflow", summary={...}, ...)
...
```

This is similar to `log_artifact`and it handles setting `kind=V1ArtifactKind.MODEL` automatically.

> **Note**: In reference lineage tracking, you can also provide extra key/value summary to augment the lineage information. `log_artifact_ref` also accepts a `kind` to specify the artifact kind.

## Saving artifacts and reference logging

Some methods support saving artifacts, i.e. moving the file/dir under outputs, and logging the lineage reference in one call:

### log artifact

```python
tracking.log_artifact(name="file", path="path/from/file.csv", kind=V1ArtifactKind.CSV)
``` 

This will both move the `file.csv` under the `run-uuid/outputs/` folder and save a lineage reference.

you can also control where under the `run-uuid/outputs/` the file should be saved using the `rel_path` argument:

```python
tracking.log_artifact(name="file", path="path/from/file.csv", kind=V1ArtifactKind.CSV, rel_path="csv_files/new/")
```

### log model

Similar behaviour can be achieved with `log_model`:

```python
tracking.log_model(name="file", path="path/from/model.h5", framework="scikit", summary={"additional_key": "value"})
``` 

This will both save the model under the `run-uuid/outputs/model/` folder and save a lineage reference.

You can change the default model sub-folder by passing a rel_path:

```python
tracking.log_model(name="file", path="path/from/model.h5", framework="scikit", summary={"additional_key": "value"}, rel_path="different_folder")
``` 

## Versioned assets logging

Versioned assets logging is useful when the user needs to save the same asset but several times in the same run, based on timestamp values and/or steps.
For that the tracking module will automatically generate a new subpath under the `assets` sub-folder, e.g. `assets/model/dirname_STEP_NUMBER` or `assets/audio/filename_STEP_NUMBER`,
and each time a logging method is called with the same artifact name, it will create a new entry in the event file and a new subpath with the step number.

Usually users should use the versioned logging when they need an easy way to explore a specific versioned artifact in the dashboards tab, 
the UI will create a widget with a step slider to load a new file version based on the step number.
 
> **Note**: some logging functions do not save assets, they just populate the event file with new values, for instance scalars/metrics/text tracking.

Example logging a custom curve several times during the runtime:

```python
tracking.log_curve("custom-curve", x_array, y_array, step=14)
...
tracking.log_curve("custom-curve", x_array, y_array, step=140)
```

Example logging a model for every checkpoint:

```python
tracking.log_model(model_path, name="model", framework="tensorflow", step=3)
...
tracking.log_model(model_path, name="model", framework="tensorflow", step=140)
```

Example logging a generic artifact several times:

```python
tracking.log_artifact(artifact_path, kind=V1ArtifactKind.DATAFRAME, name="df-pickle", step=3)
...
tracking.log_artifact(artifact_path, kind=V1ArtifactKind.DATAFRAME, name="df-pickle", step=140)
```
