Features
- Unified
- Implement all components of a distributed application in a single language
- Universal
- Freely express any distributed architecture
- Safe
- Enjoy static type-safety across components and static checks for architectural constraints
Concepts
Specify Architecture
Define the architectural relation of the components of the distributed system
@peer type Server <: {
type Tie <: Multiple[Client]
}
@peer type Client <: {
type Tie <: Single[Server]
}
Specify Placement
Control where data is located and computations are executed
val items: Items on Server =
getCurrentItems()
val ui: UI on Client =
new UI
Compose
Combine data flow across components through reactive programming
val items: Var[Items] on Server =
Var(getCurrentItems())
on[Client] {
Signal {
items.asLocal() map createUIEntry
}
}
Step-by-Step Example
The example demonstrates the design of a chat application in five steps
- Declaring a Server
- Declaring a Client
- Declaring a message event on the Client and firing the event for every user input
- Declaring a publicMessage event on the Server aggregating all events from the clients
- Declaring a client-side observer for the server-side publicMessage event
@multitier object Chat {
@peer type Server <: { type Tie <: Multiple[Client] }
}
@peer type Client <: { type Tie <: Single[Server] }
val message = on[Client] { Evt[String]() }
def main() = on[Client] {
for (line <- io.Source.stdin.getLines)
message.fire(line)
}
val publicMessage = on[Server] {
message.asLocalFromAllSeq map { case (_, message) => message }
}
publicMessage.asLocal observe println
Getting Started
Try out ScalaLoci Examples from GitHub
Add ScalaLoci to Your Project
-
Enable support for macro annotations in your build.sbt
-
for Scala 2.13
scalacOptions += "-Ymacro-annotations"
-
for Scala 2.11 or 2.12 (Macro Paradise Plugin)
addCompilerPlugin("org.scalamacros" % "paradise" % "2.1.1" cross CrossVersion.patch)
-
for Scala 2.13
-
Add the resolver for the ScalaLoci dependencies to your build.sbt
resolvers += Resolver.bintrayRepo("stg-tud", "maven")
-
Add the ScalaLoci dependencies that you need for your system to your build.sbt
-
ScalaLoci language (always required):
libraryDependencies += "de.tuda.stg" %% "scala-loci-lang" % "0.4.0"
-
Transmitter for the types of values to be accessed remotely
(built-in Scala types and standard collections are directly supported without additional dependencies)
-
REScala reactive events and signals
libraryDependencies += "de.tuda.stg" %% "scala-loci-lang-transmitter-rescala" % "0.4.0"
-
REScala reactive events and signals
-
Network communicators to connect the different components of the distributed system
-
TCP [JVM only]
libraryDependencies += "de.tuda.stg" %% "scala-loci-communicator-tcp" % "0.4.0"
-
WebSocket (using Akka HTTP on the JVM) [server: JVM only, client: JVM and JS web browser APIs]
libraryDependencies += "de.tuda.stg" %% "scala-loci-communicator-ws-akka" % "0.4.0"
-
WebSocket (Play integration) [server: JVM only, client: JVM and JS web browser APIs]
libraryDependencies += "de.tuda.stg" %% "scala-loci-communicator-ws-akka-play" % "0.4.0"
-
WebSocket (using Javalin on the JVM) [server: JVM only, client: JS web browser APIs]
libraryDependencies += "de.tuda.stg" %% "scala-loci-communicator-ws-javalin" % "0.4.0"
-
WebRTC [JS web browser APIs only]
libraryDependencies += "de.tuda.stg" %% "scala-loci-communicator-webrtc" % "0.4.0"
-
TCP [JVM only]
- Serializer for network communication
-
ScalaLoci language (always required):
Showcases
P2P Chat
The example logs messages that are sent and received by each participant as a composition of the data flow from the local UI and from remote chat partners. In the application, nodes are connected to multiple remote nodes and maintain a one-to-one chat with each. Users can select any chat to send messages. The messageSent
event is defined as subjective value filtering the ui.messageTyped
messages from the UI for the currently active chat partner node
. The messageLog
signal contains the chat log for the chat between the local peer instance and the remote node
given as parameter. It merges the remote stream for the chat messages from the remote instance node
and the local stream subjective to the remote instance node
via the ||
operator. The chat log is a signal created using list
, which extends the list by an element for each new event occurrence in the merged stream. The chatLogs
signal folds the remote[Node].joined
event stream, which is fired for each newly connected chat partner, into a signal that contains the chat logs for every chat partner generated by calling messageLog
.
@multitier object Chat {
@peer type Node <: { type Tie <: Multiple[Node] }
type SingleChatLog = Signal[List[String]]
type MultiChatLogs = Signal[List[SingleChatLog]]
val ui: UI on Node = UI()
val messageSent = on[Node] sbj { node: Remote[Node] =>
ui.messageTyped filter { _ => ui.isSelectedChat(node) }
}
def messageLog(node: Remote[Node]): Local[SingleChatLog] on Node =
((messageSent from node).asLocal ||
(messageSent to node)).list
val chatLogs: MultiChatLogs on Node =
remote[Node].joined.fold(List.empty[SingleChatLog]) { (chats, node) =>
messageLog(node) :: chats
}
}
Tweets
The example shows how the operators in a processing pipeline can be placed on different peers to count the tweets that each author produces in a tweet stream. The application receives a stream of tweets on the Input
peer, selects those containing the "multitier"
string on the Filter
peer, extracts the author
for each tweet on the Mapper
peer, and stores a signal with a map counting the tweets from each author on the Folder
peer.
@multitier object TweetAuthoring {
@peer type Input <: { type Tie <: Single[Filter] }
@peer type Filter <: { type Tie <: Single[Mapper] with Single[Input] }
@peer type Mapper <: { type Tie <: Single[Folder] with Single[Filter] }
@peer type Folder <: { type Tie <: Single[Mapper] }
val tweetStream: Event[Tweet] on Input =
retrieveTweetStream()
val filtered: Event[Tweet] on Filter =
tweetStream.asLocal filter { tweet => tweet.hasHashtag("multitier") }
val mapped: Event[Author] on Mapper =
filtered.asLocal map { tweet => tweet.author }
val folded: Signal[Map[Author, Int]] on Folder =
mapped.asLocal.fold(Map.empty[Author, Int].withDefaultValue(0)) {
(map, author) => map + (author -> (map(author) + 1))
}
}
E-Mail Application
The example shows a client–server e-mail application. The server stores a list of e-mails. The client can request the e-mails received in the n previous days containing a given word. The client user interface displays the e-mails broken into several pages. If the word is not in the current page, the user is informed. The definition of word
of type Signal[String] on Client
defines a signal carrying strings placed on the Client
peer. Thanks to multitier reactives, the client-side signal inCurrentPage
is defined by the composition of the local client-side signal word
and the remote server-side signal filteredEmails
. The latter is defined as a composition of a local signal and two remote signals.
@multitier object MailApp {
@peer type Client <: { type Tie <: Single[Server] }
@peer type Server <: { type Tie <: Single[Client] }
val word: Signal[String] on Client = /* GUI input */
val days: Signal[Int] on Client = /* GUI input */
val allEmails: Local[Signal[List[Email]]] on Server =
/* e-mail collection */
val filteredEmails: Signal[List[Email]] on Server =
Signal {
allEmails() filter { email =>
(email.date < Date.today() - days.asLocal()) &&
(email.text contains word.asLocal())
}
}
val inCurrentPage: Local[Boolean] on Client =
Signal { isCurrentFirstPage(word(), filteredEmails.asLocal()) }
Master–worker
The example shows a ScalaLoci implementation of the master–worker pattern where a master node dispatches tasks – double a number, for simplicity – to workers. The taskStream
on the master carries the tasks as events. The assocs
signal contains the assignments of workers to tasks. It folds over the taskStream || taskResult.asLocalFromAllSeq
event stream that fires for every new task (taskStream
) and every completed task (taskResult.asLocalFromAllSeq
). The assignTasks
method assigns a worker to the new task (taskAssocs
), or enqueues the task if no worker is free (taskQueue
) based on the folded event (taskChanged
) and the currently connected worker instances (remote[Worker].connected
). The deployTask
event subjectively provides every worker instance with the task it is assigned. Workers provide the result in the taskResult
event stream which the master aggregates into the result
signal. The signal is updated for every event to contain the sum of all values carried by the events.
@multitier object MasterWorker {
@peer type Master <: { type Tie <: Multiple[Worker] }
@peer type Worker <: { type Tie <: Single[Master] }
class Task(v: Int) { def exec(): Int = 2 * v }
// to add tasks: `taskStream.fire(Task(42))`
val taskStream: Local[Event[Task]] on Master = Evt[Task]()
val assocs: Local[Signal[Map[Remote[Worker], Task]]] on Master =
(taskStream || taskResult.asLocalFromAllSeq).fold(
Map.empty[Remote[Worker], Task],
List.empty[Task]) { (taskAssocs, taskQueue, taskChanged) =>
assignTasks(taskAssocs, taskQueue, taskChanged, remote[Worker].connected)
}
val deployTask: Signal[Task] per Worker on Master =
worker: Remote[Worker] => Signal{ assocs().get(worker) }
val taskResult: Event[Int] on Worker =
deployTask.asLocal.changed collect { case Some(task) => task.exec() }
val result: Signal[Int] on Master =
taskResult.asLocalFromAllSeq.fold(0) { case (acc, (worker, result)) =>
acc + result
}
}
Publications
- Daniel Sokolowski, Jan-Patrick Lehr, Christian Bischof, Guido Salvaneschi. 2020. Leveraging Hybrid Cloud HPC with Multitier Reactive Programming. In 2020 IEEE/ACM International Workshop on Interoperability of Supercomputing and Cloud Technologies, SuperCompCloud, Atlanta, GA, USA, pp. 27-32. https://doi.org/10.1109/SuperCompCloud51944.2020.00010 [definitive version] [author version]
- Pascal Weisenburger, Johannes Wirth, and Guido Salvaneschi. 2020. A Survey of Multitier Programming. ACM Computing Surveys 53, 4, Article 81 (September 2020), 35 pages. https://doi.org/10.1145/3397495 [definitive version] [author draft version]
- Pascal Weisenburger and Guido Salvaneschi. 2020. Implementing a Language for Distributed Systems: Choices and Experiences with Type Level and Macro Programming in Scala. The Art, Science, and Engineering of Programming, Volume 4, Issue 3, Article 17 (February 2020), 29 pages. http://doi.org/10.22152/programming-journal.org/2020/4/17 [definitive version] [author version]
- Daniel Sokolowski, Philipp Martens, and Guido Salvaneschi. 2019. Multitier Reactive Programming in High Performance Computing. In Proceedings of the 6th ACM SIGPLAN International Workshop on Reactive and Event-Based Languages and Systems, REBLS. October 20–25, 2019, Athens, Greece. [author version]
- Pascal Weisenburger and Guido Salvaneschi. 2019. Multitier Modules. In Proceedings of the 33rd European Conference on Object-Oriented Programming, ECOOP, Leibniz International Proceedings in Informatics (LIPIcs). July 15–19, 2019, London, United Kingdom. Schloss Dagstuhl – Leibniz-Zentrum für Informatik, Dagstuhl, Germany. http://doi.org/10.4230/LIPIcs.ECOOP.2019.3 [definitive version] [author version] [PDF slides] [HTML slides] [video recording]
- Pascal Weisenburger and Guido Salvaneschi. 2019. Tutorial: Developing Distributed Systems with Multitier Programming. In Proceedings of the 13th ACM International Conference on Distributed and Event-based Systems, DEBS. June 24–28, 2019, Darmstadt, Germany. ACM, New York, NY, USA, 203–204. http://doi.org/10.1145/3328905.3332465 [definitive version] [author version]
- Pascal Weisenburger, Mirko Köhler, and Guido Salvaneschi. 2018. Distributed System Development with ScalaLoci. Proceedings of the ACM on Programming Languages 2, OOPSLA, Article 129 (October 2018), 30 pages. http://doi.org/10.1145/3276499 [definitive version] [author version] [PDF slides] [HTML slides] [video recording]
- Pascal Weisenburger, Tobias Reinhard, and Guido Salvaneschi. 2018. Static Latency Tracking with Placement Types. In Companion Proceedings for the ISSTA/ECOOP 2018 Workshops (FTfJP’18). July 16–21, 2018, Amsterdam, Netherlands. ACM, New York, NY, USA, 34–36. http://doi.org/10.1145/3236454.3236486 [definitive version] [author version]
- Pascal Weisenburger. 2016. Multitier Reactive Abstractions. In Companion Proceedings of the 2016 ACM SIGPLAN International Conference on Systems, Programming, Languages and Applications: Software for Humanity (SPLASH Companion 2016). October 30 – November 4, 2016, Amsterdam, Netherlands. ACM, New York, NY, USA, 18–20. http://doi.org/10.1145/2984043.2984051 [definitive version] [author version]