Features
- Multitier Programming
- Implement all components of a distributed application in a single language
- Universal Placement
- Freely express any distributed architecture through placement types
- Safety Guarantees
- Enjoy static type-safety across components and static checks for architectural constraints
Concepts
Specify Architecture
Define the architectural relation of the components of the distributed system
@peer type Server <: {
type Tie <: Multiple[Client]
}
@peer type Client <: {
type Tie <: Single[Server]
}
Specify Placement
Control where data is located and computations are executed
val items: Items on Server =
getCurrentItems()
val ui: UI on Client =
new UI
Compose
Combine data flow across components through reactive programming
val items: Var[Items] on Server =
Var(getCurrentItems())
on[Client] {
Signal {
items.asLocal() map createUIEntry
}
}
Step-by-Step Example
The example demonstrates the design of a chat application in five steps
- Declaring a Server
- Declaring a Client
- Declaring a message event on the Client and firing the event for every user input
- Declaring a publicMessage event on the Server aggregating all events from the clients
- Declaring a client-side observer for the server-side publicMessage event
Getting Started
Try out ScalaLoci Examples from GitHub
Add ScalaLoci to Your Project
-
Enable support for macro annotations in your build.sbt:
-
for Scala 2.13
scalacOptions += "-Ymacro-annotations"
-
for Scala 2.11 or 2.12 (Macro Paradise Plugin)
addCompilerPlugin("org.scalamacros" % "paradise" % "2.1.1" cross CrossVersion.patch)
-
for Scala 2.13
-
Add the ScalaLoci dependencies that you need for your system to your build.sbt:
-
ScalaLoci language (always required)
libraryDependencies ++= Seq( "io.github.scala-loci" %% "scala-loci-language" % "0.5.0" % "compile-internal", "io.github.scala-loci" %% "scala-loci-language-runtime" % "0.5.0")
-
Transmitter for the types of values to be accessed remotely
(built-in Scala types and standard collections are directly supported without additional dependencies)
-
REScala reactive events and signals
libraryDependencies += "io.github.scala-loci" %% "scala-loci-language-transmitter-rescala" % "0.5.0"
-
REScala reactive events and signals
-
Network communicators to connect the different components of the distributed system
-
TCP [JVM only]
libraryDependencies += "io.github.scala-loci" %% "scala-loci-communicator-tcp" % "0.5.0"
-
WebSocket (using web browser APIs) [JS only, client only]
libraryDependencies += "io.github.scala-loci" %% "scala-loci-communicator-ws-webnative" % "0.5.0"
-
WebSocket (using Akka HTTP) [JVM only]
libraryDependencies += "io.github.scala-loci" %% "scala-loci-communicator-ws-akka" % "0.5.0"
-
WebSocket (Play integration using Akka HTTP) [JVM only]
libraryDependencies += "io.github.scala-loci" %% "scala-loci-communicator-ws-akka-play" % "0.5.0"
-
WebSocket (using Javalin) [JVM only, server only]
libraryDependencies += "io.github.scala-loci" %% "scala-loci-communicator-ws-javalin" % "0.5.0"
-
WebSocket (using Jetty) [JVM only]
libraryDependencies += "io.github.scala-loci" %% "scala-loci-communicator-ws-jetty" % "0.5.0"
-
WebRTC (using web browser APIs) [JS only]
libraryDependencies += "io.github.scala-loci" %% "scala-loci-communicator-webrtc" % "0.5.0"
-
TCP [JVM only]
-
Serializer for network communication
-
µPickle serialization
libraryDependencies += "io.github.scala-loci" %% "scala-loci-serializer-upickle" % "0.5.0"
-
Circe serialization
libraryDependencies += "io.github.scala-loci" %% "scala-loci-serializer-circe" % "0.5.0"
-
Jsoniter Scala serialization
libraryDependencies += "io.github.scala-loci" %% "scala-loci-serializer-jsoniter-scala" % "0.5.0"
-
µPickle serialization
-
ScalaLoci language (always required)
Showcases
Chat
The example logs messages that are sent and received by each participant as a composition of the data flow from the local UI and from remote chat partners. In the application, nodes are connected to multiple remote nodes and maintain a one-to-one chat with each. Users can select any chat to send messages. The messageSent
event is defined as subjective value filtering the ui.messageTyped
messages from the UI for the currently active chat partner node
. The messageLog
signal contains the chat log for the chat between the local peer instance and the remote node
given as parameter. It merges the remote stream for the chat messages from the remote instance node
and the local stream subjective to the remote instance node
via the ||
operator. The chat log is a signal created using list
, which extends the list by an element for each new event occurrence in the merged stream. The chatLogs
signal folds the remote[Node].joined
event stream, which is fired for each newly connected chat partner, into a signal that contains the chat logs for every chat partner generated by calling messageLog
.
@multitier object Chat {
@peer type Node <: { type Tie <: Multiple[Node] }
type SingleChatLog = Signal[List[String]]
type MultiChatLogs = Signal[List[SingleChatLog]]
val ui: UI on Node = UI()
val messageSent = on[Node] sbj { node: Remote[Node] =>
ui.messageTyped filter { _ => ui.isSelectedChat(node) }
}
def messageLog(node: Remote[Node]): Local[SingleChatLog] on Node =
((messageSent from node).asLocal ||
(messageSent to node)).list
val chatLogs: MultiChatLogs on Node =
remote[Node].joined.fold(List.empty[SingleChatLog]) { (chats, node) =>
messageLog(node) :: chats
}
}
Tweets
The example shows how the operators in a processing pipeline can be placed on different peers to count the tweets that each author produces in a tweet stream. The application receives a stream of tweets on the Input
peer, selects those containing the "multitier"
string on the Filter
peer, extracts the author
for each tweet on the Mapper
peer, and stores a signal with a map counting the tweets from each author on the Folder
peer.
@multitier object TweetAuthoring {
@peer type Input lt;: { type Tie <: Single[Filter] }
@peer type Filter <: { type Tie <: Single[Mapper] with Single[Input] }
@peer type Mapper <: { type Tie <: Single[Folder] with Single[Filter] }
@peer type Folder <: { type Tie <: Single[Mapper] }
val tweetStream: Event[Tweet] on Input =
retrieveTweetStream()
val filtered: Event[Tweet] on Filter =
tweetStream.asLocal filter { tweet => tweet.hasHashtag("multitier") }
val mapped: Event[Author] on Mapper =
filtered.asLocal map { tweet => tweet.author }
val folded: Signal[Map[Author, Int]] on Folder =
mapped.asLocal.fold(Map.empty[Author, Int].withDefaultValue(0)) { (map, author) =>
map + (author -> (map(author) + 1))
}
}
Master–worker
The example shows a ScalaLoci implementation of the master–worker pattern where a master node dispatches tasks – double a number, for simplicity – to workers. The taskStream
on the master carries the tasks as events. The assocs
signal contains the assignments of workers to tasks. It folds over the taskStream || taskResult.asLocalFromAllSeq
event stream that fires for every new task (taskStream
) and every completed task (taskResult.asLocalFromAllSeq
). The assignTasks
method assigns a worker to the new task (taskAssocs
), or enqueues the task if no worker is free (taskQueue
) based on the folded event (taskChanged
) and the currently connected worker instances (remote[Worker].connected
). The deployTask
event subjectively provides every worker instance with the task it is assigned. Workers provide the result in the taskResult
event stream which the master aggregates into the result
signal. The signal is updated for every event to contain the sum of all values carried by the events.
@multitier object MasterWorker {
@peer type Master <: { type Tie <: Multiple[Worker] }
@peer type Worker <: { type Tie <: Single[Master] }
class Task(v: Int) { def exec(): Int = 2 * v }
// to add tasks: `taskStream.fire(Task(42))`
val taskStream: Local[Event[Task]] on Master = Evt[Task]()
val assocs: Local[Signal[Map[Remote[Worker], Task]]] on Master =
(taskStream || taskResult.asLocalFromAllSeq).fold(
Map.empty[Remote[Worker], Task],
List.empty[Task]) { (taskAssocs, taskQueue, taskChanged) =>
assignTasks(taskAssocs, taskQueue, taskChanged, remote[Worker].connected)
}
val deployTask: Signal[Task] per Worker on Master = { worker: Remote[Worker] =>
Signal { assocs().get(worker) }
}
val taskResult: Event[Int] on Worker =
deployTask.asLocal.changed collect { case Some(task) => task.exec() }
val result: Signal[Int] on Master =
taskResult.asLocalFromAllSeq.fold(0) { case (acc, (worker, result)) =>
acc + result
}
}
Token Ring
The example models a token ring, where every node in the ring can send a token for another node. Multiple tokens can circulate in the ring simultaneously until they reach their destination. Every node has exactly one predecessor and one successor. We define a Prev
and a Next
peer and specify that a Node
itself is both a predecessor and a successor and has a single tie to its own predecessor and a single tie to its successor. Tokens are passed from predecessors to successors, hence nodes access the tokens sent from their predecessor. For this reason, values are placed on the Prev
peer. Every node has a unique ID. The sendToken
event sends a token along the ring to another peer instance. The recv
event stream provides the data received by each peer instance. Each node fires recv
when it receives a token addressed to itself, i.e., when the receiver
equals the node ID and forwards other tokens. The expression sent.asLocal \ recv
evaluates to an event stream of all events from sent.asLocal
for which recv
does not fire. Merging such stream (of forwarded tokens) with the sendToken
stream via the ||
operator injects both new and forwarded tokens into the ring.
@multitier object TokenRing {
@peer type Prev <: { type Tie <: Single[Prev] }
@peer type Next <: { type Tie <: Single[Next] }
@peer type Node <: Prev with Next { type Tie <: Single[Prev] with Single[Next] }
val id: Id on Prev = Id()
val sendToken: Local[Event[(Id, Token)]] on Prev = Evt[(Id, Token)]()
val recv: Local[Event[Token]] on Prev =
sent.asLocal collect {
case (receiver, token) if receiver == id => token
}
val sent: Event[(Id, Token)] on Prev =
(sent.asLocal \ recv) || sendToken
}
Publications
- George Zakhour, Pascal Weisenburger, Guido Salvaneschi. 2023. Type-Safe Dynamic Placement with First-Class Placed Values. Proceedings of the ACM on Programming Languages 7, OOPSLA2, Article 297 (October 2023), 29 pages. https://doi.org/10.1145/3622873 [definitive version] [author version] [PDF slides] [video recording]
- Saverio Giallorenzo, Fabrizio Montesi, Marco Peressotti, David Richter, Guido Salvaneschi, Pascal Weisenburger. 2021. Multiparty Languages: The Choreographic and Multitier Cases. In Proceedings of the 35th European Conference on Object-Oriented Programming, ECOOP, Leibniz International Proceedings in Informatics (LIPIcs). July 11–17, 2021, Aarhus, Denmark. Schloss Dagstuhl – Leibniz-Zentrum für Informatik, Dagstuhl, Germany. https://doi.org/10.4230/LIPIcs.ECOOP.2021.22 [definitive version] [author version] [PDF slides] [video recording]
- Daniel Sokolowski, Jan-Patrick Lehr, Christian Bischof, Guido Salvaneschi. 2020. Leveraging Hybrid Cloud HPC with Multitier Reactive Programming. In Proceedings of the 3rd IEEE/ACM International Workshop on Interoperability of Supercomputing and Cloud Technologies, SuperCompCloud. November 11, 2020, Atlanta, GA, USA. IEEE, Piscataway, NY, USA, 27–32. https://doi.org/10.1109/SuperCompCloud51944.2020.00010 [definitive version] [author version]
- Pascal Weisenburger, Johannes Wirth, Guido Salvaneschi. 2020. A Survey of Multitier Programming. ACM Computing Surveys, Volume 53, Issue 4, Article 81 (September 2020), 35 pages. https://doi.org/10.1145/3397495 [definitive version] [author version]
- Pascal Weisenburger, Guido Salvaneschi. 2020. Implementing a Language for Distributed Systems: Choices and Experiences with Type Level and Macro Programming in Scala. The Art, Science, and Engineering of Programming, Volume 4, Issue 3, Article 17 (February 2020), 29 pages. https://doi.org/10.22152/programming-journal.org/2020/4/17 [definitive version] [author version]
- Daniel Sokolowski, Philipp Martens, and Guido Salvaneschi. 2019. Multitier Reactive Programming in High Performance Computing. In Proceedings of the 6th ACM SIGPLAN International Workshop on Reactive and Event-Based Languages and Systems, REBLS. October 20–25, 2019, Athens, Greece. [author version]
- Pascal Weisenburger, Guido Salvaneschi. 2019. Multitier Modules. In Proceedings of the 33rd European Conference on Object-Oriented Programming, ECOOP, Leibniz International Proceedings in Informatics (LIPIcs). July 15–19, 2019, London, United Kingdom. Schloss Dagstuhl – Leibniz-Zentrum für Informatik, Dagstuhl, Germany. https://doi.org/10.4230/LIPIcs.ECOOP.2019.3 [definitive version] [author version] [PDF slides] [HTML slides] [video recording]
- Pascal Weisenburger, Guido Salvaneschi. 2019. Tutorial: Developing Distributed Systems with Multitier Programming. In Proceedings of the 13th ACM International Conference on Distributed and Event-based Systems, DEBS. June 24–28, 2019, Darmstadt, Germany. ACM, New York, NY, USA, 203–204. https://doi.org/10.1145/3328905.3332465 [definitive version] [author version]
- Pascal Weisenburger, Mirko Köhler, Guido Salvaneschi. 2018. Distributed System Development with ScalaLoci. Proceedings of the ACM on Programming Languages 2, OOPSLA, Article 129 (October 2018), 30 pages. https://doi.org/10.1145/3276499 [definitive version] [author version] [PDF slides] [HTML slides] [video recording]
- Pascal Weisenburger, Tobias Reinhard, Guido Salvaneschi. 2018. Static Latency Tracking with Placement Types. In Companion Proceedings for the ISSTA/ECOOP 2018 Workshops (FTfJP’18). July 16–21, 2018, Amsterdam, Netherlands. ACM, New York, NY, USA, 34–36. https://doi.org/10.1145/3236454.3236486 [definitive version] [author version]
- Pascal Weisenburger. 2016. Multitier Reactive Abstractions. In Companion Proceedings of the 2016 ACM SIGPLAN International Conference on Systems, Programming, Languages and Applications: Software for Humanity (SPLASH Companion 2016). October 30 – November 4, 2016, Amsterdam, Netherlands. ACM, New York, NY, USA, 18–20. https://doi.org/10.1145/2984043.2984051 [definitive version] [author version]