Your

Home Page

Path: Home

We supply middleware solutions for cost-effective, efficient, selective, scalable, and near real-time database replication. We call it database distribution.

What is real-time database distribution?

Real-time DB distribution is the creation and maintenance of dynamic partial copies of a central database on many client-machines. Each client defines its view (the portion of the database seen by the client). Updates on the database are distributed to all interested clients in near realtime performance. Clients can change their view at any time. The distributed data is delivered to application software running on the client machines. Clients can store their partial copy of the database in memory, or in a local database, or directly in user-interface components. Real-time database distribution can be considered as a generalization of real-time database replication.

Common problems in real-time data-distribution:

  • Clients have different views on the database.
  • Network topology is constantly changing.
    New clients can join the network and existing client can leave the network, at any time.
  • Clients need to change their subscription.
  • Some clients are connected over low-bandwidth communications, while others are connected over high-bandwidth communications.
  • Data must be sent to each site only once.

In what ways are our products efficient?

Our data-distribution products send binary and not self-describing messages over the network. Only modified fields are sent through http connections, with very little overhead. We distribute only the delta, which means the log of all operations performed on the data. We do this without compromising transactional-integrity and guaranteed-delivery.

We support multi-site topology, in which each delta is sent only once to each site (and only when it is of interest to at least one client in the site). Our proxy servers receive incoming distribution from their mother-site and serve all clients in their local site.

We support concurrent and fast loads from our servers and our proxies to clients, with zero overhead on the database. Three kinds of loads are supported:
  • Full loads to new joining clients.
  • Partial loads to existing clients when clients change their subscription.
  • Sync loads to clients rejoining after being disconnected for a long time.
Snapshots are downloaded to the client utilizing http streaming mechanism (Transfer-encoding = chunked). Thus, the total load time is the time required to pass the data through the communication lines, or the time required by the client to process received data (when the client is slower than the communications speed).

When a client reconnects to the distribution servers after a short disconnection period (a few minutes) no load is required, and the client can immediately resume its reception of steady-state distribution.

What is selective distribution?

Each entity is distributed only to clients, which are permitted to see it, and in addition choose to select it.

At the server-side: A list of topics (each topic is a string, whose format is a convention between the server and the clients) is attached to each hierarchical entity in the database. Business logics can assign topics to created entities, and are also allowed to update the topics-list of existing entities.

Each client: defines a topics-slice to express its interest in the entities selected by the slice. A client can change its subscription (topics-slice) whenever it needs to.

The meaning of topics:
Each topic belongs to one of two kinds of topics:
  • The name of a client or a group of clients. When a topic of that kind is attached to an entity, the entity is distributed to the specified client or group of clients. It is assumed that each client specifies in its topics-slice the client-name and the names of all groups of clients it is a member of.
  • A name representing an attribute of the entity. This enables clients to select all entities having the attributes that are of interest to them.

How near are we to real-time database distribution?

The distribution products support both poll-mode in which each client periodically (for example once every 2 seconds) polls the server for incoming distribution, and push-mode in which the servers push deltas to all relevant clients immediately after modifications are committed to the database. Push-mode enables the distribution of updates to all interested clients in tens of milliseconds following database commit time.

Scalability

The distribution servers run on a cluster of machines with linear scalability since each machine can service any client's request. Machines can be added to the cluster on the fly.

The distribution servers can run on multi-processor machines, and fully utilize all processors. To achieve this a very efficient locking algorithm is used, in which the internal data-structures are not read-write locked for long time-periods.

A cost effective solution

The distribution server machines backup each other. Each delta-server (responsible for steady-state distribution) supports hundreds of connected clients. Each cache-server (responsible for snapshot loads) can handle tens of load-requests concurrently.

The distribution servers (both delta-server and cache-servers) run on low-cost blade machines. The cost of each server machine is approximately $1500.
DB to DB distribution
use-case:

Many to many:
In this model, the database is distributed over a set of database servers. The products are used for handling distribution between server machines.

Flexible topology:
Each database server is directly connected to some other database servers, and it distributes its local database to all directly connected database servers.

Selective:
Each database-server defines a topics-slice per each directly connected database-server for selecting entities from the other database-server.

No master:
There is no master server. Each database server is allowed to modify all entities (including their topics-lists) at any time.

Changing topology:
The set of database servers which are directly connected to a specific database server can change at anytime. Dynamic changes of topology are possible due to fast synchronizations and the registration mechanism.

A proven technology:

Our software is used by mission critical systems!

Source-code included:

We sell LiveDist(C#) and LiveDist(Java) as open-source. Thus, our customers have no long run dependency on us.

Internals course:

Customers who wish to self-maintain LiveDist(C#) or LiveDist(Java), can order internal courses for their software engineers. These courses assist them to fully understand the internal behavior of the distribution products.

DB to clients distribution
use-case:

Full-load:
When a client first joins the network, it defines a topics-slice. This causes a frozen snapshot of the database, containing all database entities selected by the slice, to be downloaded to the client

Gap-closing:
The database continues to process transactions during the snapshot download period. This causes a gap to open between the client and the database. Following the snapshot load, the client receives a burst of deltas, in order to close the gap.

Steady-state:
After closing the gap, the client enters steady state, in which it receives deltas whenever entities that are selected by the topics-slice are inserted into, updated in, or deleted from the database.

Partial-load:
The client can change its subscription, by redefining its topics-slice. This causes a partial-snapshot to be loaded to the client, containing all entities not selected by the old slice but selected by the new slice


The distribution products:

LiveDist(Java) and LiveDist(C#) are similar products, running on different platforms.

LiveDist(Java):
A Java product (J2EE). Supported databases: Oracle.

LiveDist(C#):
A C# product (.Net). Supported databases: SQL-Server, Oracle.

Read more...