etl4s

Operators

Operator Name What it does
~> Chain a ~> b - output of a feeds into b
& / &> Fan-out a & b - run both with the same input (&> runs them concurrently)
* / *> Product a * b - run on different inputs (*> runs them concurrently)
>> Sequence a >> b - run in order, keep b's result
| Fan-in a | b - route an Either input to the matching branch
+ Choice a + b - route an Either input through independent branches
<|> Fallback a <|> b - if a throws, run b on the same input

Chain nodes with ~>

a b c

The output of one node feeds the next. This is the backbone of most pipelines:

val checkout = 
     parseCart ~> applyTax ~> total

Send one input to several nodes with & / &>

b c (b, c)

Both nodes get the same input and their results come back as a pair. Use &> to run them concurrently under an effect like Future. Here one user goes in and both cards are built from it:

val dashboard = 
     fetchUser ~> (profileCard & recentOrders) ~> layout

Give each node its own input with * / *>

b d (b, d)

Where & sends the same input to both, * splits a pair: the first half goes left, the second goes right. Use *> to run them concurrently. Here the name goes to trimName and the age to bumpAge:

val register = 
     (trimName * bumpAge) ~> saveAccount

Run steps in order with >>

a b discarded b's result

Runs each step in order and keeps only the last result. Handy for setup steps you run for their side effects:

val ingest = 
     clearStaging >> warmCache >> (extract ~> transform ~> load)

Route an Either to the matching branch with |

b Left c Right out

A Left goes to the left node, a Right to the right one, and both produce the same type. Here a legacy id or a uuid comes in, and one account comes out:

val load = 
     (byLegacyId | byUuid) ~> loadAccount

Handle each side of an Either with +

b Left c Right Left' Right'

Like |, but each side keeps its own output type, so you still have an Either at the end. Here refunds and charges are handled separately, then merged back with |:

val settle = 
     classifyTxn ~> (handleRefund + handleCharge) ~> (fileRefund | postCharge)

Fall back to another node with <|>

a b on throw result

Runs the left node, and if it throws, runs the right one on the original input. Here fetchCached only runs if fetchLive fails:

val convert = 
     (fetchLive <|> fetchCached) ~> applyRates ~> total

Under an effect that tracks errors (like Try), the recovery goes through the effect instead of a thrown exception.

Concurrency needs a concurrent effect

&> and *> only run their branches concurrently when compiled to a concurrent effect such as Future (or IO), via .compile[Future]. Plain unsafeRun (the Id interpreter) has no threads, so it runs them in sequence.

Auto-flatten and .zip

Chaining fan-outs auto-flattens the tuple: a & b & c has type Node[X, (A, B, C)] (not ((A, B), C)), and the same holds for &>:

import etl4s._

val n1 = Node[String, Int](_.length)
val n2 = Node[String, String](_.toUpperCase)
val n3 = Node[String, Boolean](_.nonEmpty)

val flat = n1 & n2 & n3
flat.unsafeRun("hi")
flat has type Node[String, (Int, String, Boolean)], and you will get:
(2, "HI", true)
If you already have a node whose output is a nested tuple, .zip flattens it:
val nested  = (n1 & n2) & n3
val flatten = nested.zip
nested has type Node[String, ((Int, String), Boolean)] and flatten has type Node[String, (Int, String, Boolean)].