etl4s
API Reference
  • .lineage(name, inputs, outputs, ...)
  • .toDot
  • .toMermaid
  • .toJson
  • .lineageName(name)
  • .lineageInputs(inputs*)
  • .lineageOutputs(outputs*)
  • .withLineage(lineage)

Diagrams

.toDot and .toMermaid draw any etl4s pipeline, in two ways:

Both work the same on config-aware nodes.

Draw a single pipeline

Any pipeline can draw itself. Take this one:

import etl4s._

val extract5 = Node(5)
val double   = Node[Int, Int](_ * 2)
val triple   = Node[Int, Int](_ * 3)
val combine  = Node[(Int, Int), Int] { case (a, b) => a + b }
val saveToDb = Node[Int, Unit](n => println(s"saved $n"))

val pipeline =
     extract5 ~> (double & triple) ~> combine ~> saveToDb

pipeline.toDot
extract5 Int double combine Int triple Int Int Int saveToDb Int Unit Any

showTypes = false hides the type labels, and direction changes the layout (LR by default, or TB, RL, BT):

pipeline.toDot(showTypes = false)
pipeline.toMermaid(direction = Direction.TB)

To get the same structure as a list, see .stages in Your First Pipeline.

Draw lineage across pipelines

To show datasources, schedules and dependencies between pipelines, tag your nodes with .lineage, then call .toDot, .toMermaid or .toJson on a Seq of them.

import etl4s._

val A = Node[String, String](identity)
  .lineage(
    name = "A",
    inputs = List("s1", "s2"),
    outputs = List("s3"), 
    schedule = "0 */2 * * *"
  )

val B = Node[String, String](identity)
  .lineage(
    name = "B",
    inputs = List("s3"),
    outputs = List("s4", "s5")
  )

Export them to JSON, DOT (Graphviz) or Mermaid:

Seq(A, B).toJson
Seq(A, B).toDot
Seq(A, B).toMermaid

Export the diagram

DOT

For Graphviz:

Seq(A, B).toDot

Mermaid

Seq(A, B).toMermaid
graph LR
    classDef pipeline fill:#e1f5fe,stroke:#01579b,stroke-width:2px,color:#000
    classDef dataSource fill:#f3e5f5,stroke:#4a148c,stroke-width:2px,color:#000

    A["A<br/>(0 */2 * * *)"]
    B["B"]
    s1(["s1"])
    s2(["s2"])
    s3(["s3"])
    s4(["s4"])
    s5(["s5"])

    s1 --> A
    s2 --> A
    A --> s3
    s3 --> B
    B --> s4
    B --> s5
    A -.-> B
    linkStyle 6 stroke:#ff6b35,stroke-width:2px

    class A pipeline
    class B pipeline
    class s1 dataSource
    class s2 dataSource
    class s3 dataSource
    class s4 dataSource
    class s5 dataSource

The orange dotted arrows are inferred dependencies.

JSON

Seq(A, B).toJson

The JSON has three top-level keys:

Lineage parameters

Parameter Description Default
name Unique identifier required
inputs Input data sources empty
outputs Output data sources empty
upstreams Explicit dependencies (Node, Reader, or String) empty
schedule Human-readable schedule, e.g. 0 */2 * * * none
cluster Group name for related pipelines none
description Free-text description ""
group Logical grouping label ""
tags List[String] of arbitrary tags empty
links Map[String, String] of label -> URL empty

.lineage works the same on config-aware nodes.

Set fields one at a time

To add metadata bit by bit, use .lineageName, .lineageInputs and .lineageOutputs, or .withLineage to attach a whole Lineage at once.

Note that .toJson always gives you the lineage metadata, even on a single node.

Declare dependencies yourself

Use upstreams when one pipeline depends on another without sharing data. Add a node C:

val C = Node[String, String](identity)
  .lineage("C", upstreams = List(A, B))

Then draw all three:

Seq(A, B, C).toDot

C now depends on A and B, even though it doesn't read their outputs.

Group pipelines into clusters

Give related pipelines the same cluster and they're drawn together:

val B = Node[String, String](identity)
  .lineage(
    name = "B",
    inputs = List("s3"),
    outputs = List("s4", "s5"),
    cluster = "Y"
  )

val C = Node[String, String](identity)
  .lineage(
    name = "C",
    upstreams = List(A, B),
    cluster = "Y"
  )

Seq(A, B, C).toDot