Replication service allows maintaining backup copies of Persistence API storages.
Replication service uses Aeron transport for transmitting data to backup instances. Replication can work in synchronous and asynchronous mode. In synchronous mode, storage sends notification about every operation to backups and waits for acknowledgment back. It blocks calling thread till receiving acknowledgment or till predefined timeout will expire.
Replication service consists of 2 parts: leader and backup. Depends on instance role it needs to initialize and start one or another instance.
Leader instance is responsible for
notifications about exist and new storages
delivering notifications about storage’s operations
handling synchronization requests
Backup instance is responsible for
creating required storages on backup
update storages state (process notifications about storage’s operations)
It is very useful to use replication with
ClusterManager allows to start and stop leader or backup replication instance depends on node role in cluster. Also
ClusterManager can notify replication service about available nodes and they address automatically.
To use replication service with Cluster Manager it needs to implement
LocalNodeBackupListener to receive and handle notifications about cluster state. You can use default implementations of these listeners for replication service. They will handle all events and enable and configure appropriate service depends on node role in cluster.
Initialize replication leader
- REPLICATION_LEADER_PORT - port, which leader is listening for synchronization request from backups.
- REPLICATION_BROADCAST_PORT - port, which each backup node is listening for replication data from the leader.
Initialize replication backup
Creating storage directly
Creating storage with Persistence API
Configuration of replication service there is in 2 configuration files: replication service configuration (replication.properties) and Aeron Media Driver configuration (aeron.properties).
Replication Service configuration
Replication Service configuration there is in replication.properties.
Default (initial) replication mode (synchronous or asynchronous)
Default (initial) timeout for synchronous replication in milliseconds. Process could be blocked for this timeout till receive acknowledgment from another side.
Default: 0 milliseconds (async mode)
The size of the leader incoming ring buffer must be power of 2.+ Default: 512 bytes
The wait strategy to use for the leader incoming ring buffer (see Disruptor WaitStrategy).
The size of the leader outgoing ring buffer, must be power of 2.
Default: 2048 bytes
The wait strategy to use for the leader outgoing ring buffer (see Disruptor WaitStrategy).
The size of the backup incoming ring buffer, must be power of 2.
Default: 1024 bytes
The wait strategy to use for the backup incoming ring buffer (see Disruptor WaitStrategy).
The size of the backup outgoing ring buffer, must be power of 2.
The wait strategy to use for the backup outgoing ring buffer (see Disruptor WaitStrategy).
Use embedded aeron media driver (see Aeron Embedded Media Driver).
Provides an IdleStrategy for the thread responsible for communicating with the Aeron Media Driver (see Aeron Idle Strategies).
Aeron Media Driver configuration
Please find description of Aeron configuration options at official page.
Replication service lifecycle
The goal of replication service is maintaining of full copies of the storages within a network. To do this it should support at least 2 operations:
synchronization of data to restore actual state
replication data in runtime to maintain the state
The service can replicate data in two modes - synchronous and asynchronous.
Sending messages asynchronously does not expect confirmation about the successful receipt and processing by another side. Replication is performed in parallel with the thread, which changes the data in storages.
Asynchronous replication expects to receive confirmation about the successful processing, at least, from one of the backup nodes. A processing timeout can be specified for the storage instance during its creating (otherwise the default settings will be used). The thread that changes the data in storages, is blocked until receiving acknowledgment or until expiring of the timeout. In the last case, the warning, that signals that the data was not successfully transmitted to any of the backup nodes during the specified period, will be logged.
Replication mode is set separately for each of the repositories, so it is possible to simultaneously use both the storage with synchronous replication and with asynchronous.
Each backup node maintains two transport channels because Aeron transport is unidirectional by its nature:
incoming: for sending synchronization requests and synchronous acknowledgments
outgoing: for receiving information about changing the data in storages
The listening port is assigned to each node of replication service, depends on its role. If you change the node’s role, the listening port is not changed. One and the same port will be always listened by leader and backups always know how to contact it. Same for the port that is listening by backups.
Persistent storages are designed for incremental updates. Internal storage contains log of operations like 'APPEND' and 'REMOVE'. In case of synchronization or replication data, it needs only to send new updates.
Data synchronization procedure
After starting the backup instance synchronizes its state with the leader.
The synchronization procedure is:
The backup sends
GET_ALL_RESOURCESrequest to the leader right after start.
The leader sends lists of existing queries and sequences in response as 'RESOURCE_LIST' messages.
CorrelationIdis also sent for every storage.
CorrelationIdis unique id for storage. It uses in the next communications.
The backup loads exist and create non-exist storages.
The backup sends
SYNC_REQrequest for every storage and passes its index. The timestamp of the last reset is passed for a sequence in addition.
The leader sends
SYNC_REQ_ACCEPTEDmessage in answer. It compares indexes and timestamps and resent all required data. For sequence, it can also send
SEQUENCE_RESETmessage, if reset was missed. At the end, the leader sends
SYNC_FINISHEDmessage to indicate the border of synchronization answer.
Data replication procedure
On every operation with persistent storage, the leader sends updates to all backups.
The replication procedure is:
On adding the new item to internal storage the leader sends
SEQUENCE_APPENDmessage. New data and internal ordered index are sent with this message.
The backup received this message and compares the expected index with received.
If received index is different then expected, it starts synchronization procedure (see Data synchronization procedure, n.4)
If the leader indicates that this is synchronous storage and it expects acknowledgment, the backup sends back
FIX session replication
FIX Antenna Java also can use replication storages thru Persistence API.
To enable replicated storage for FIX session it needs to setup replication leader and backup (Initialize replication leader, Initialize replication backup) and use next configuration options for FIX antenna (in
storageFactoryproperty. This factory allows to use Persistence API for storing FIX session state.
PersistenceEnableFactoryis based on
FilesystemStorageFactoryand delegate all operation Persistence API objects. To work with this API it requires implementation of
PersistentFactoryBuilder. The last one should construct instance of
ReplicatedPersistenceFactoryBuilderlike a factory builder for
PersistentFactoryBuilderand build replicated instance of factory. It uses
replicationTimeoutoptions from FIX antenna config (or from session’s
Configurationinstance) to configure synchronous or asynchronous replication more for FIX session.
- Define replication timeout in milliseconds. Zero value for this option will enable asynchronous replication for FIX session.