Resource Generation
nagi compile automatically generates Nagi resources from a dbt project. This page describes the generated resources and merge behavior.
All examples below assume an Origin named my-project.
Mapping
Origin reads manifest.json from the dbt project and generates Nagi resources using the following mapping.
| dbt resource | Generated resource |
|---|---|
| dbt source | kind: Asset |
| dbt model | kind: Asset |
| dbt test | kind: Conditions |
Generated Asset names are prefixed with the Origin name: {origin}.{model}. See kind: Origin — Asset Naming for details.
dbt source
A dbt source is converted to an Asset resource.
From the Origin configuration, connection and autoSync are applied. upstreams is not set. dbt source tags are converted to metadata.labels with the dbt/ prefix.
yaml
apiVersion: nagi.io/v1alpha1
kind: Asset
metadata:
name: my-project.raw-sales
spec:
connection: my-bigquery
autoSync: true
onDrift:
- conditions: dbt-tests-my-project.raw-sales
sync: nagi-skip-sync
dbt model
A dbt model is converted to an Asset resource.
From the Origin configuration, connection and autoSync are applied.
The dbt lineage (dependencies between dbt models and from dbt source to dbt model) is converted to spec.upstreams, and dbt model tags are converted to metadata.labels with the dbt/ prefix.
yaml
apiVersion: nagi.io/v1alpha1
kind: Asset
metadata:
name: my-project.daily-sales
labels:
dbt/finance: ""
dbt/daily: ""
spec:
connection: my-bigquery
autoSync: true
upstreams:
- my-project.raw-sales
onDrift:
- conditions: dbt-tests-my-project.daily-sales
sync: my-project-dbt-run
dbt test
A dbt test is converted to a Conditions resource.
The generated Conditions is set as an onDrift entry on the Asset corresponding to the dbt model or dbt source that the dbt test targets.
yaml
apiVersion: nagi.io/v1alpha1
kind: Conditions
metadata:
name: dbt-tests-my-project.daily-sales
spec:
- name: dbt-test-daily-sales
type: Command
run: [dbt, test, --select, daily_sales]
Drift Detection and Sync
When drift from the desired state is detected (Drifted), the Sync that Nagi executes differs depending on the dbt resource.
Drift on dbt source
A dbt source is data ingested from an external system, so Nagi and dbt cannot repair it.
In this case, no Sync is executed, and processing of downstream Assets that depend on this dbt source is blocked until the data is repaired externally.
This behavior uses nagi-skip-sync (a built-in Sync that does nothing and exits successfully).
Drift on dbt model
Nagi executes dbt run --select <model> to rebuild the data.
Origin automatically generates a Sync named {origin}-dbt-run (e.g. my-project-dbt-run) for this behavior. The Sync uses {{ asset.modelName }} to pass the original dbt model name (without the Origin prefix).
yaml
kind: Sync
metadata:
name: my-project-dbt-run
spec:
run:
type: Command
args: ["dbt", "run", "--select", "{{ asset.modelName }}", "--project-dir", "../dbt-project", "--profile", "my_project", "--target", "dev"]
You can replace this Sync with a different one using the Origin's defaultSync field.
Merge with User-defined Resources
You may want to add desired state definitions to an Origin-generated Asset. If you define a kind: Asset with the same name in resources/, the onDrift lists of both are concatenated. Merging only works for kind: Asset.
Merge target
Only onDrift is subject to merging. All other fields retain the Origin's values.
Evaluation order
onDrift entries are evaluated from top to bottom, and the Sync of the first entry that becomes Drifted is executed.
You can specify the merge method by setting mergePosition on the onDrift entries of the user-defined Asset.
beforeOrigin: Before Origin entries (default)afterOrigin: After Origin entries
Name conflicts
If resources other than Asset (Connection, Conditions, Sync) have duplicate names, a compile error occurs.
Example
This example shows how to add a Freshness desired state to the dbt model daily_sales to monitor data freshness. Assume daily_sales is a daily aggregation table updated by a morning dbt run.
1. Origin-generated resources
The Conditions and Asset automatically generated by Origin.
```yaml
target/conditions/dbt-tests-my-project.daily-sales.yaml (Origin-generated)
apiVersion: nagi.io/v1alpha1 kind: Conditions metadata: name: dbt-tests-my-project.daily-sales spec: - name: dbt-test-daily-sales type: Command run: [dbt, test, --select, daily_sales] ```
```yaml
target/assets/my-project.daily-sales.yaml (Origin-generated)
apiVersion: nagi.io/v1alpha1 kind: Asset metadata: name: my-project.daily-sales labels: dbt/finance: "" dbt/daily: "" spec: connection: my-bigquery upstreams: - my-project.raw-sales onDrift: - conditions: dbt-tests-my-project.daily-sales sync: my-project-dbt-run ```
2. User-defined resources
Since this table is updated every morning, it is checked once a day, and if it has not been updated for more than 25 hours, it is rebuilt with dbt run (24 hours + 1 hour buffer for job execution time).
```yaml
resources/freshness-daily-sales.yaml (user-defined)
apiVersion: nagi.io/v1alpha1 kind: Conditions metadata: name: freshness-daily-sales spec: - name: updated-within-25h type: Freshness maxAge: 25h interval: 24h ```
```yaml
resources/daily-sales.yaml (user-defined)
apiVersion: nagi.io/v1alpha1 kind: Asset metadata: name: my-project.daily-sales spec: onDrift: - conditions: freshness-daily-sales sync: my-project-dbt-run # mergePosition: afterOrigin ```
3. Compiled result
beforeOrigin
With the default (beforeOrigin), user-defined entries are placed before Origin entries. Since Freshness is evaluated first, if the data is older than 25 hours, dbt run is executed regardless of the dbt test results.
```yaml
target/assets/my-project.daily-sales.yaml (after compile)
apiVersion: nagi.io/v1alpha1 kind: Asset metadata: name: my-project.daily-sales labels: dbt/finance: "" dbt/daily: "" spec: connection: my-bigquery upstreams: - my-project.raw-sales onDrift: - conditions: freshness-daily-sales sync: my-project-dbt-run - conditions: dbt-tests-my-project.daily-sales sync: my-project-dbt-run ```
afterOrigin
When mergePosition: afterOrigin is specified, user-defined entries are placed after Origin entries. The dbt test is evaluated first, and Freshness is evaluated only if the test passes.
```yaml
resources/freshness-daily-sales.yaml (afterOrigin specified)
apiVersion: nagi.io/v1alpha1 kind: Asset metadata: name: my-project.daily-sales spec: onDrift: - conditions: freshness-daily-sales sync: my-project-dbt-run mergePosition: afterOrigin ```
```yaml
target/assets/my-project.daily-sales.yaml (after compile)
apiVersion: nagi.io/v1alpha1 kind: Asset metadata: name: my-project.daily-sales labels: dbt/finance: "" dbt/daily: "" spec: connection: my-bigquery upstreams: - my-project.raw-sales onDrift: - conditions: dbt-tests-my-project.daily-sales sync: my-project-dbt-run - conditions: freshness-daily-sales sync: my-project-dbt-run ```