etl4s
API Reference
  • .requires[T]
  • .provide(env)
  • .provideContext(env)
  • Reader[T, Node]
  • Etl4sCtx[T]

Configuration

Pipelines usually need config: database URLs, API keys, thresholds. You don't want to thread it through every function, and you want to swap it out in tests.

Declare what a node needs with .requires, then .provide it once at the top. Under the hood it's just a Reader (Config => Node[In, Out]) with some syntax.

import etl4s._

case class Cfg(key: String)

val readData   = Node("data")
val tagWithKey = Node[String, String].requires[Cfg] { cfg => data =>
  s"${cfg.key}: $data"
}

val pipeline = 
     readData ~> tagWithKey

pipeline.provide(Cfg("secret")).unsafeRun()
You will get:
"secret: data"

Keep config out of your types

Some libraries add config as an extra type parameter, which here would look like Node[Cfg, In, Out]. etl4s doesn't. A node stays Node[In, Out], and .requires[Cfg] wraps only the nodes that need config.

The operators don't care which is which:

val load = Node[Unit, String](_ => "data")
val tag: Reader[Cfg, Node[String, String]] =
  Node[String, String].requires[Cfg] { c => s => c.key + s }

load ~> tag

~>, &, >> and friends mix plain and config-aware nodes, work out what the whole pipeline needs, and leave you one .provide at the end.

Combine configs from different nodes

Split your config into traits, and etl4s works out what the pipeline needs:

trait HasDb { def dbUrl: String }
trait HasAuth { def apiKey: String }
case class AppConfig(dbUrl: String, apiKey: String) extends HasDb with HasAuth

val save = Node[String, Unit].requires[HasDb] { cfg => data =>
  println(s"Saving to ${cfg.dbUrl}: $data")
}

val fetch = Node[Unit, String].requires[HasAuth] { cfg => _ =>
  s"Fetched with ${cfg.apiKey}"
}

val toUpper = Node[String, String](_.toUpperCase)


val pipeline = 
     fetch ~> toUpper ~> save

pipeline.provide(AppConfig("jdbc:pg", "secret-key")).unsafeRun()

.requires[T] turns a node into a Reader[T, Node].

How requirements combine

Three rules decide what a pipeline ends up needing.

1. Plain nodes add nothing. A plain node needs no config, so the pipeline only asks for what the config-aware nodes need:

val fetch   = Node[Unit, String].requires[HasAuth] { c => _ => s"got ${c.apiKey}" }
val toUpper = Node[String, String](_.toUpperCase)

val p = 
     fetch ~> toUpper

p.provide(AppConfig("jdbc:pg", "key")).unsafeRun()
HasAuth fetch ~> toUpper = HasAuth fetch ~> toUpper

2. Different configs combine. If one node needs HasAuth and another needs HasDb, the pipeline needs HasAuth & HasDb. You provide one value that is both, like the AppConfig above.

HasAuth HasDb

3. The more specific type wins. If one node needs HasDb and another needs AppConfig (which extends HasDb), the pipeline just needs AppConfig.

HasDb AppConfig

Group config-driven nodes with Etl4sCtx

Etl4sCtx[T] lets you put nodes that share a config in one module:

case class DbConfig(url: String, timeout: Int)

object DataPipeline extends Etl4sCtx[DbConfig] {

  val fetch = Etl4sCtx.Extract[Unit, String] { cfg => _ =>
    s"Connected to ${cfg.url} with timeout ${cfg.timeout}s"
  }

  val save = Etl4sCtx.Load[String, Unit] { cfg => data =>
    println(s"Saving to ${cfg.url}: $data")
  }

  val pipeline = fetch ~> save
}

DataPipeline.pipeline.provide(DbConfig("jdbc:pg", 5000)).unsafeRun()

Scala 2

Spell out the types to help inference:

Transform.requires[Config, String, String] { cfg => input =>
  cfg.key + input
}
In Scala 3, the preferred syntax is:
Transform[String, String].requires[Config] { cfg => input =>
  cfg.key + input
}