Dimdwarf Application Server

Dimdwarf is an unintrusive distributed application server for Java. You write single-threaded, event-driven POJO code - the server makes it multi-threaded, persisted and transactional. The primary target audience is developers of online multiplayer games. The programming model is the same as on Project Darkstar (now RedDwarf). Dimdwarf applications can be deployed on Darkstar with minimal modifications and vice versa.

Table of Contents
How does it work?
Articles about Dimdwarf
Project Goals
How can I participate?
Comparison of Darkstar and Dimdwarf

How does it work?

Applications are divided into short tasks, preferably only some milliseconds. For example when a client sends a message to the server, the server starts a new task in which that message is handled. Tasks may also be scheduled for execution by other tasks. The server executes multiple tasks concurrently, but from the application developer's point of view each task is single-threaded and he needs to do no concurrency management - the server takes care of concurrency.

Each task is a transactional unit of work. When entity objects are accessed during a task, they are loaded and deserialized from the database, and when the task ends they are serialized and the modifications are written back into the database. If network messages are sent during a task, those will be sent only after the transaction commits. Likewise when a task schedules new tasks, they will be executed only after the task has committed.

In case a task fails because of transaction conflict, the task is rolled back and retried. The tasks must have no side effects (creating new threads, accessing the file system, modifying static fields etc.), so the user will not know if a task was retried. The server may use some heuristics to avoid transaction conflicts, but in the end the developer needs to organize his application so that there are no entities which are modified concurrently by many unrelated tasks, which might be a performance bottleneck.

Application classes are divided into entities and value objects. An entity has an identity which spans many tasks - an entity equals only itself. A value object is only a value and there is no distinction between a value object and a copy of the value object - a value object equals another value object if they both contain the same data. In Dimdwarf, entities are marked with an @Entity annotation. Both entities and value objects must be Serializable.

When an entity is serialized into database, the graph of all value objects which are referred by the entity are serialized together with the entity. When an entity refers to other entities, those references are replaced with a transparent reference proxy (an object with the same external interface as the target entity, but inside it has only the database ID of the target entity). To achieve concurrency, the application classes need to be divided into entities of suitable size, so that loading an entity and its value object graph from database is quick, and that not many concurrent tasks modify the same entities.

This programming model is basically the same as in Project Darkstar. Some terminology may differ (Dimdwarf talks about entities, Darkstar uses managed objects). Also Darkstar exposes managed references to the application programmer, but Dimdwarf hides them by default. Darkstar supports clustering, but Dimdwarf does not (yet) support it. You may refer to Darkstar's documentation for more details on the architecture.

Articles about Dimdwarf

Let's Code Dimdwarf 2010-09-22 by Esko Luontola
Screencast series of developing Dimdwarf. The first episode explains Dimdwarf's new architecture (with pictures!).

New Architecture for Dimdwarf 2009-06-11 by Esko Luontola
Overview of the planned scalable architecture for Dimdwarf.

Introduction to Dimdwarf 2009-05-07 by Esko Luontola
Background history and project goals explained.

Project Darkstar's New World of Online Games: A Conversation With Jeff Kesselman 2007-10 by Janice J. Heiss
An article explaining the motivation behind Project Darkstar, the system which inspired Dimdwarf. Darkstar and Dimdwarf have the same goals, but a different implementation.

Project Goals

A lightweight embeddable unintrusive test-friendly application server which takes care of transactions, persistence, task scheduling and networking. Compatible with Project Darkstar.

Writing applications for Dimdwarf should be as similar to writing plain Java code as possible - i.e. in Java objects created with new will remain in memory until they are not used, after which they are garbage collected. Likewise on Dimdwarf, objects created with new will remain in database until they are not used, after which they are garbage collected. Accessing persisted objects in database shall be no different from accessing normal Java objects in memory.

At least in the short term, the server will not support clustering. At most multi-threading on a single JVM is supported. To allow the applications to scale, a Dimdwarf-to-Darkstar compatibility adapter shall be created, so that the same application can be deployed both on Dimdwarf and on Project Darkstar (whose main goal is massive scalability). This compatibility implies that Dimdwarf's features will be ported to Darkstar, if Darkstar does not yet have a particular feature (e.g. garbage collection).

The framework should be as unintrusive and test-friendly as possible. Ideally the application code should have no dependencies to the framework. Writing unit tests for application code should be easy, without any initialization code for the framework. Mocking the framework should be possible in less than 100 lines of code (for example with a single-threaded ScheduledExecutorService and some networking API mocks).

The target audience of Dimdwarf is primarily game developers (the same as Darkstar) and secondarily business application developers. For online games (including MMO games) it is preferrable to optimize for worst-case latency and consistency, at the expense of throughput and durability. The long term goals of Dimdwarf include: support for database refactoring, querying the database, rolling upgrades (of both applications and servers), high availability and scalability ("Dimdwarf-HA").

How can I participate?

Join the mailing list and tell others about your ideas. You can begin contributing right away by cloning the project's Git repository at github.com/orfjackal/dimdwarf. Please follow GitHub's instructions about forking a project and submitting your modifications. Then send to the mailing list a message telling about your changes, so that they can be pulled into the main repository.

The guidelines for code changes are:

  1. The code follows Simple Design (also see XPE page 57 and Essential XP: Emergent Design), i.e. in order of importance:
    1. Runs all the tests
    2. Contains no duplication
    3. Expresses the intent of the programmer
    4. Minimizes the number of classes and methods
  2. The tests are written just before the production code (i.e. use TDD)
  3. The tests are organized as executable specifications of the system's behaviour (i.e. use BDD)
  4. Follow good object-oriented design, for example by applying the SOLID principles

This will allow us to keep the code moldable - keep the software soft.

Some good books to read are Clean Code: A Handbook of Agile Software Craftsmanship and Growing Object-Oriented Software, Guided by Tests. The code in this project should measure up to the code in those books. There are links to more learning resources at www.orfjackal.net/devlinks

Comparison of Darkstar and Dimdwarf

Dimdwarf was inspired by Darkstar and it implements the same programming model, so obviously there are many similarities between the two systems. But there are also quite many differences, which are summarized in the following table:

Application API
Darkstar exposes its ManagedReferences to the application programmer, which results in tight coupling between an application using Darkstar and the API of Darkstar. Running Darkstar applications outside Darkstar requires using MockSGS or a similar mocking framework. Dimdwarf hides the implementation details of its entities using transparent references. Dimdwarf applications can easily be run outside Dimdwarf without an extensive mocking framework.
Darkstar applications must delete their entities manually from the database. Some of the public API even requires the deletion of objects, for example a ClientSession is closed by removing it from the database. Dimdwarf relies on database garbage collection in everything. It is simply not possible to delete objects from the database manually.
Darkstar applications get access to the API through a static AppContext class, using the service locator pattern. Since Darkstar 0.9.8, it has been possible to change the InternalContext that backs a AppContext to make unit testing Darkstar applications possible. Dimdwarf applications get access to the API through dependency injection. Unit testing Dimdwarf applications does not require any custom test setup and teardown.
Server Architecture
Darkstar uses pessimistic locking to handle concurrent data access. The transaction isolation level is serializable. Dimdwarf uses optimistic locking to handle concurrent data access. The transaction isolation level snapshot isolation.
Darkstar forces a time limit on tasks. The default is 100ms, but tasks are expected to be much shorter. Dimdwarf is not planned to have task time limits. Long-running tasks have a higher probability of conflicting with other tasks, if they modify the same objects, but well isolated long-running tasks are possible. The server may use optimizations that give lower priority to long-running tasks.
Many of Darkstar's core services are used concurrently and need to be thread-safe. They use lock-based synchronization. The server nodes use Java RMI to communicate with each other. Darkstar's planned architecture uses each component single-threadedly, so they don't need to be thread-safe. Message-passing is used to communicate between components inside a server and with other server nodes.
Darkstar's multi-node version is planned to use a central database server together with a write cache on each node. Dimdwarf's multi-node version is planned to use a shared-nothing architecture, where the database is distributed over all nodes.
In Darkstar, the tasks of a client are run on the same node where the client is connected. It is planned that to move a client to another node, the client needs to co-operate and connect to that node. In Dimdwarf, it is planned that tasks can be relocated on a per-task basis. The clients connect to a gateway node, which forwards the messages from the client to a backend node for execution. The backend can be changed without the client knowing about it.
Darkstar uses Berkeley DB as its database. Dimdwarf uses a custom in-memory database. It is planned to persist the database to disk asynchronously.
License
Darkstar uses GPL and its server API uses GPL with the classpath exception. Distributing the server-side of an application together with Darkstar requires the application being licensed under GPL. Dimdwarf uses the Apache License 2.0 for everything. You can freely embed Dimdwarf in a commercial closed-source application. (Dimdwarf's dependencies use MIT, BSD or Apache License.)
Development Process
Darkstar's tests (latest trunk on 2008-12-05) take over 17 minutes to run and only 2367 of 2378 tests pass. The tests do very much I/O, but even having the database files on a RAM drive does not speed up the tests enough. Developing Darkstar using TDD is painful, because it is not practical to run all tests after every change. Dimdwarf's test are FIRST. As of 2008-12-05 it takes about 1.4 seconds to run all the 461 tests (on a C2Q6600 @ 3.0GHz). All tests pass and they are CPU bound. Dimdwarf is developed using TDD.
If you want to contribute code to Darkstar, you need to register at java.net, sign a Sun Contributor Agreement, request write access to the central Subversion repository and wait for a reply before you can commit. Or then you can copy the latest revision from the central Subversion repository, set up your own repository on another server, and manually keep the repositories in sync. If you want to contribute code to Dimdwarf, you need to install Git and run the command git clone git://github.com/orfjackal/dimdwarf.git (or fork it in GitHub), after which you can begin committing right away to your local repository. Then when you have produced something to be included in the project, contact Dimdwarf's developers and ask for them to pull from your repository or send them a patch. Merging and keeping repositories in sync is easy with Git.
Darkstar's development process requires every change to be reviewed by someone else before it is committed to the main branch. Dimdwarf's development process requires all tests to pass before committing changes to the main branch.
Darkstar is developed by a team of developers who are financed by Sun Microsystems. Its development progresses at a steady pace.
UPDATE: As of 2010-02-02, due to Sun being bought by Oracle, Sun Labs engineering effort is no longer being applied to Darkstar development.
Dimdwarf is developed by a single developer on his spare time (if you join, then we are two :-). Its development progresses irregularly.
Code Style
Darkstar's code uses extensive Javadoc comments for just about every class, method and field. Dimdwarf's code uses descriptive names instead of extensive Javadocs. Comments are used only to explain the reasons for non-intuitive code. Tests and code are the primary low-level documentation. The behaviour of the system is defined by executable specifications (i.e. tests) and the intentions of the programmer are documented in code and tests, so that they will not get out of sync with reality (like free-form text documentation does). High-level documentation will be written after the implementation is ready - when it is known what became of the system.
In Darkstar, the typical class is very large. On 2009-05-21, the server module has 181 classes of which 25% are <90 LOC, 50% are <180 LOC, 75% are <300 LOC, and 14 classes are over 1000 LOC, the longest file being 1943 LOC. In Dimdwarf, the typical class is very small and is focused on a single responsibility (SRP). On 2009-05-21, the core module has 156 classes of which 75% are <50 LOC, and 15 classes are over 100 LOC, the longest file being 194 LOC.
Darkstar's components use the service locator pattern to fetch their dependencies. To find out the dependencies of a class, you need to read the body of its constructor (for example the constructor of DataServiceImpl is over 100 lines long - longer than most classes in Dimdwarf!). Mocking dependencies is hard. Dimdwarf's components use the dependency injection pattern to receive their dependencies. To find out the dependencies of a class, you need to read the signature of its constructor. Mocking dependencies is easy.

Get Dimdwarf Application Server at SourceForge.net. Fast, secure and Free Open Source software downloads