-
.requires[T] -
.provide(env) -
.provideContext(env) -
Reader[T, Node] -
Etl4sCtx[T]
Configuration
Pipelines usually need config: database URLs, API keys, thresholds. You don't want to thread it through every function, and you want to swap it out in tests.
Declare what a node needs with .requires, then .provide it once at the top. Under the hood it's just a Reader
(Config => Node[In, Out]) with some syntax.
import etl4s._
case class Cfg(key: String)
val readData = Node("data")
val tagWithKey = Node[String, String].requires[Cfg] { cfg => data =>
s"${cfg.key}: $data"
}
val pipeline =
readData ~> tagWithKey
pipeline.provide(Cfg("secret")).unsafeRun()
Keep config out of your types
Some libraries add config as an extra type parameter, which here would look like Node[Cfg, In, Out]. etl4s
doesn't. A node stays Node[In, Out], and .requires[Cfg] wraps only the nodes that need config.
The operators don't care which is which:
val load = Node[Unit, String](_ => "data")
val tag: Reader[Cfg, Node[String, String]] =
Node[String, String].requires[Cfg] { c => s => c.key + s }
load ~> tag
~>, &, >> and friends mix plain and config-aware nodes, work out what the whole pipeline needs, and leave
you one .provide at the end.
Combine configs from different nodes
Split your config into traits, and etl4s works out what the pipeline needs:
trait HasDb { def dbUrl: String }
trait HasAuth { def apiKey: String }
case class AppConfig(dbUrl: String, apiKey: String) extends HasDb with HasAuth
val save = Node[String, Unit].requires[HasDb] { cfg => data =>
println(s"Saving to ${cfg.dbUrl}: $data")
}
val fetch = Node[Unit, String].requires[HasAuth] { cfg => _ =>
s"Fetched with ${cfg.apiKey}"
}
val toUpper = Node[String, String](_.toUpperCase)
val pipeline =
fetch ~> toUpper ~> save
pipeline.provide(AppConfig("jdbc:pg", "secret-key")).unsafeRun()
.requires[T] turns a node into a Reader[T, Node].
How requirements combine
Three rules decide what a pipeline ends up needing.
1. Plain nodes add nothing. A plain node needs no config, so the pipeline only asks for what the config-aware nodes need:
val fetch = Node[Unit, String].requires[HasAuth] { c => _ => s"got ${c.apiKey}" }
val toUpper = Node[String, String](_.toUpperCase)
val p =
fetch ~> toUpper
p.provide(AppConfig("jdbc:pg", "key")).unsafeRun()
2. Different configs combine. If one node needs HasAuth and another needs HasDb, the pipeline needs
HasAuth & HasDb. You provide one value that is both, like the AppConfig above.
3. The more specific type wins. If one node needs HasDb and another needs AppConfig (which extends
HasDb), the pipeline just needs AppConfig.
Group config-driven nodes with Etl4sCtx
Etl4sCtx[T] lets you put nodes that share a config in one module:
case class DbConfig(url: String, timeout: Int)
object DataPipeline extends Etl4sCtx[DbConfig] {
val fetch = Etl4sCtx.Extract[Unit, String] { cfg => _ =>
s"Connected to ${cfg.url} with timeout ${cfg.timeout}s"
}
val save = Etl4sCtx.Load[String, Unit] { cfg => data =>
println(s"Saving to ${cfg.url}: $data")
}
val pipeline = fetch ~> save
}
DataPipeline.pipeline.provide(DbConfig("jdbc:pg", 5000)).unsafeRun()
etl4s