Page tree
Skip to end of metadata
Go to start of metadata

  * The following instructions are for CentOS/RedHat 7. They don't work for CentOS/RedHat 6.

Deployment scheme

In general, the cluster might look like this (

Shared storage might be a SAN-attached device, Fibre channel attached device, or TCP/IP attached device. The device might be attached to all nodes simultaneously, or cluster resource manager can attach it to the active node only. The device might be in turn a resilient one, presenting the distributed file system with software or hardware replication between filesystem nodes. In the case of a geographically distributed cluster, the shared storage also can be distributed geographically in the same way as cluster nodes, and cluster resource manager can attach to the node the storage instance in the same geo location.

Typical FIXEdge cluster setup is the two-node failover cluster with shared storage:


In this setup, the TCP/IP attached device is used. A two nodes GlusterFS file server with mirroring is used as the shared device. The device is mounted to both nodes simultaneously. See the diagram below:


Setting up GlusterFS Server 

Reference articles:

Install software

Do these steps on both servers: and

Download and install
$ sudo wget -P /etc/yum.repos.d
$ sudo yum install glusterfs
$ sudo yum install glusterfs-fuse
$ sudo yum install glusterfs-server
Check installed version
$ glusterfsd --version

glusterfs 3.6.2 built on Jan 22 2015 12:58:10
Repository revision: git://
Copyright (c) 2006-2013 Red Hat, Inc. <>
It is licensed to you under your choice of the GNU Lesser
General Public License, version 3 or any later version (LGPLv3
or later), or the GNU General Public License, version 2 (GPLv2),
in all cases as published by the Free Software Foundation.
Start glusterfs services on all servers with enable them to start automatically on startup
$ sudo /etc/init.d/glusterd start
$ sudo chkconfig glusterfsd on


Do these steps only on one server. For example, on

add EVUAKYISD105D to the trusted storage pool
$ sudo gluster peer probe
Now we can check the status
$ sudo gluster peer status
Number of Peers: 1
Uuid: 04126ea4-a0e5-4854-a38e-69c24d9c05aa
State: Peer in Cluster (Connected)
On both nodes create a folder to be shared
$ cd /
$ sudo mkdir data
$ sudo chmod a+rwx /data
Create the storage share and start it
$ sudo gluster volume create clusterdata replica 2 transport tcp force

volume create: clusterdata: success: please start the volume to access data
$ sudo gluster volume start clusterdata

volume start: clusterdata: success
$ sudo gluster volume info

Volume Name: clusterdata
Type: Replicate
Volume ID: fa31c2de-f7d6-4f05-97e5-e310d2680d94
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: tcp

Mounting GlusterFS storage on FIXEdge cluster nodes

These commands shall be run on both FIXEdge nodes: and

Install client components
$ sudo wget -P /etc/yum.repos.d
$ sudo yum install glusterfs-client
Mount remote storage
$ sudo mkdir /data
$ sudo chmod a+rwx /data
$ sudo mount.glusterfs /data
$ mount
/dev/mapper/VolGroup00-LogVol00 on / type ext4 (rw,noatime,nodiratime,noacl,commit=60,errors=remount-ro)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
tmpfs on /dev/shm type tmpfs (rw)
/dev/sda1 on /boot type ext4 (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw) on /data type fuse.glusterfs (rw,default_permissions,allow_other,max_read=131072)

And now add the following line to /etc/fstab to ensure mounting after reboot:        /data   glusterfs       defaults,_netdev        0 0

Setting up FIXEdge instances

Copy FixEdge- to /home/user to both nodes: and

On the Node1

Unzip. Untar.

$ gunzip FixEdge-
$ tar xf FixEdge-

Move installation folder to /home/user/FixEdge:

$ cd FixEdge
$ cd v.
$ mv * /home/user/FIXEdge
$ cd ..
$ rmdir v.

Upload licenses (engine.license and fixaj2-license.bin) to /home/user/FIXEdge folder.

Move FIXEdge instance folder to shared storage:

mv FixEdge1 /data/

Edit scripts in /home/user/FixEdge/bin to direct them to the new FIXEdge instance location:

  • replace ".." with "/data".

Edit FIXICC Agent configuration:

  • /home/user/FixEdge/fixicc-agent/conf/wrapper.conf
    • = ${wrapper_home}
  • /home/user/FixEdge/fixicc-agent/conf/ 
    • EngineProperty = /data/FixEdge1/conf/
    • FIXEdgeFileSettings = /data/FixEdge1/conf/
    • LogUrl = /data/FixEdge1/log

Edit FIXEdge and engine configuration:

  • /data/FixEdge1/conf/ 
    • EngineRoot = /data
    • LicenseFile = /home/user/FixEdge/engine.license
  • /data/FixEdge1/conf/
    • FIXEdge.RootDir = /data/FixEdge1
    • Log.File.RootDir = /data/FixEdge1
    • TransportLayer.SmtpTA.DllName = /home/user/FixEdge/bin/

Install and Start FIXICC Agent daemon:

$ cd /home/user/FixEdge/fixicc-agent/bin
$ ./
$ ./

Now everything is ready to run FIXEdge on Node 1.

Prepare to copy the installation to Node 2:

$ cd /home/user
$ tar cvf FixEdge.tar FixEdge
$ gzip FixEdge.tar

Copy file FixEdge.tar.gz to Node2:/user/home

On the Node2

Unzip, untar

$ cd /home/user
$ gunzip FixEdge.tar.gz
$ tar xf FixEdge.tar

Install and Start FIXICC Agent daemon:

$ cd /home/user/FixEdge/fixicc-agent/bin
$ ./
$ ./

Open ports on both nodes

8005 for FIXICC Agent and 8901 for FIX Engine

$ sudo iptables -I INPUT -p tcp -m state --state NEW -m tcp --dport 8005 -j ACCEPT
$ sudo iptables -I INPUT -p tcp -m state --state NEW -m tcp --dport 8901 -j ACCEPT
$ sudo service iptables save

Configuring a failover cluster for FIXEdge with Pacemaker, Corosync & PCS

Reference articles:

On both nodes install needed software

$ sudo yum install corosync pcs pacemaker 

On both nodes set the password for hacluster user ('epmc-cmcc' was used)

$ sudo passwd hacluster

Configure Firewall on both nodes to allow cluster traffic:

$ sudo iptables -I INPUT -m state --state NEW -p udp -m multiport --dports 5404,5405 -j ACCEPT
$ sudo iptables -I INPUT -p tcp -m state --state NEW -m tcp --dport 2224 -j ACCEPT
$ sudo iptables -I INPUT -p igmp -j ACCEPT
$ sudo iptables -I INPUT -m addrtype --dst-type MULTICAST -j ACCEPT
$ sudo service iptables save

Start the pcsd service on both nodes

$ sudo systemctl start pcsd

From now on all commands needs to be executed on one node only. We can control the cluster by using PCS from on of the nodes.

Since we will configure all nodes from one point, we need to authenticate on all nodes before we are allowed to change the configuration. Use the previously configured hacluster user and password to do this:

$ sudo pcs cluster auth
Username: hacluster
Password: Authorized Authorized

Create the cluster and add nodes. This command command creates the cluster node configuration in /etc/corosync.conf.

$ sudo pcs cluster setup --name fixedge_cluster
Shutting down pacemaker/corosync services...
Redirecting to /bin/systemctl stop  pacemaker.service
Redirecting to /bin/systemctl stop  corosync.service
Killing any remaining services...
Removing all cluster configuration files... Succeeded Succeeded

We can start cluster now:

$ sudo pcs cluster start --all Starting Cluster... Starting Cluster...

We can check cluster status:

$ sudo pcs status cluster
Cluster Status:
 Last updated: Tue Jan 27 22:11:15 2015
 Last change: Tue Jan 27 22:10:48 2015 via crmd on
 Stack: corosync
 Current DC: (1) - partition with quorum
 Version: 1.1.10-32.el7_0.1-368c726
 2 Nodes configured
 0 Resources configured

$ sudo pcs status nodes
Pacemaker Nodes:
$ sudo corosync-cmapctl | grep members (u64) = 0 (str) = r(0) ip( (u32) = 1 (str) = joined (u64) = 0 (str) = r(0) ip( (u32) = 1 (str) = joined

$ sudo pcs status corosync
Membership information
    Nodeid      Votes Name
         1          1 (local)
         2          1

Disable the STONITH option as we don't have STONITH devices in our demo virtual environment:

$ sudo pcs property set stonith-enabled=false

For two-nodes cluster we must disable the quorum:

$ sudo pcs property set no-quorum-policy=ignore
$ sudo pcs property
Cluster Properties:
 cluster-infrastructure: corosync
 dc-version: 1.1.10-32.el7_0.1-368c726
 no-quorum-policy: ignore
 stonith-enabled: false

Add Virtual IP as a resource to the cluster:

$ sudo pcs resource create virtual_ip ocf:heartbeat:IPaddr2 ip= cidr_netmask=32 op monitor interval=30s
$ sudo pcs status resources
 virtual_ip (ocf::heartbeat:IPaddr2): Started

Add FIXEdge as a resource to cluster:

$ sudo pcs resource create FIXEdge ocf:heartbeat:anything params binfile="/home/user/FixEdge/bin/FIXEdge" cmdline_options="/data/FixEdge1/conf/" user="user" logfile="/home/user/FIXEdge_resource.log" errlogfile="/home/user/FIXEdge_resource_error.log"

For some reason in the /usr/lib/ocf/resource.d/ of the installed cluster there are many missing agents, including ocf:heartbeat:anything. You need to modify the original version (which you can download here: to make it working. The working version of the agent is attached.

This file should be copied to /usr/lib/ocf/resource.d/ and make executable:

$ sudo cp anything /usr/lib/ocf/resource.d/heartbeat/
$ sudo chmod a+rwx /usr/lib/ocf/resource.d/heartbeat/anything  

Also, to make this agent works the following lines shall be added to sudoers file:

$ sudo visudo
Defaults    !requiretty
user    ALL=(user)      NOPASSWD: ALL
root    ALL=(user)      NOPASSWD: ALL

In order to make sure that the Virtual IP and FIXEdge always stay together, we can add a constraint:

$ sudo pcs constraint colocation add FIXEdge virtual_ip INFINITY

To avoid the situation where the FIXEdge would start before the virtual IP is started or owned by a certain node, we need to add another constraint which determines the order of availability of both resources:

$ sudo pcs constraint order virtual_ip then FIXEdge
Adding virtual_ip FIXEdge (kind: Mandatory) (Options: first-action=start then-action=start)

After configuring the cluster with the correct constraints, restart it and check the status:

$ sudo pcs cluster stop --all && sudo pcs cluster start --all Stopping Cluster... Stopping Cluster... Starting Cluster... Starting Cluster...

Cluster configuration is now completed.

FIXEdge Recovery Time Objective and Recovery Point Objective

The current article describes the Recovery Time Objective (RTO) и Recovery Point Objective(RPO) for Disaster recovery in case of active-passive Cluster configuration (see FIXEdge Failover Cluster installation).

Recovery Time Objective (RTO)

  • Some functions are more important than others and may need a quicker recovery.
  • The period of time within which systems, applications, or functions must be recovered after an outage is referred to as Recovery Time Objectives (RTOs).
  • All business functions, applications or vendors that require a business continuity plan must have an RTO.

Recovery Point Objective(RPO)

  • An RPO is the maximum allowable amount of data loss an organization can tolerate as a result of an event which causes systems recovery (the transfer of work) to an alternative system environment or location.
  • An RPO is applicable only to applications that store data and are evaluated using similar to RTO requirements.
  • The RPO is not the time it takes to restore data as part of recovery. The data restoration time is part of the RTO. 

Possible RTO Values are described as "Greater than X time, up to and including Y time"

Possible RPO Values are described as "Recent sync point: Greater than X time, up to and including Y time"

Failover scenario #1. Application or cluster node failure

Failure of software or hardware within a node (e.g. application failure, node unavailability, resource exhausting). Failover Cluster consists of several application nodes. In case of one node fails – the application (or DB) is moved to another node and start there within approximately 2 minutes.

The session recovery procedure happens automatically. The missing messages should be recovered with a resend request procedure automatically. 

Session recovery requires reset sequence in case of damaged storage. The messages from the beginning of the day will be lost when resetting the sequence. See Recovery procedure for a session with corrupted storages

It is possible to manually request missing messages since sequence number equal one using the Logon (A) with the desired sequence and the subsequent Request Request (2) messages (see the FIX specification

Achievable RTO Values: "Greater than 0 seconds, up to and including 2 minutes"

Achievable RPO Values: "Recent synch point: Greater than 0 seconds, up to and including few seconds"

Failover scenario #2. Whole cluster failure

Disaster recovery environment should expect the connection. In case of disaster, the production environment is moved to DR. 

Achievable RTO Values: "Greater than 0 seconds, up to and including few seconds"

Achievable RPO Values: "Recent synch point: Greater than 0 seconds, up to and including 1 day, i.e. since the start of the day". 

The better RPO values can be achieved with additional configuration or tools like log replication tool

This procedure requires additional configuration for each client individually.


  • No labels