Page tree

Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

To service multiple FIX sessions reliablyFIXEdge can be deployed in a Linux HA cluster (Red Hat High-Availability Add-on Overview)   with 2 or 3 nodes and shared storage for keeping sessions a session's state.

The possible configurable options of shared state are:

Cluster The cluster solution utilizes Corosync and Pacemaker – tools that facilitate the HA Linux cluster for applications that do not have native support for clustering.

The This article describes the Active-Passive / Hot-Warm Failover Cluster. 

  • Active or Hot node - it is a working FIXEdge node getting runtime messages
  • Passive or Warm node - it is a FIXEdge node getting periodic updates from the an active one, and ready to start if the main one will failfails.

Virtual IP

FIXEdge nodes use virtual IP as an entry point for FIX Clients and remote administrating tools (for example FIXICC). Virtual IP is assigned to the node when it is active.

This approach gives a working working, highly-available solution and works well for active-passive clusters with few nodes and static FIX session configuration.

...

Health checks are used to figure out if a node or an application that runs on the node is operating properly.

The simplest way to do a node health check is to monitor the FIXEdge PID file.

...

Shared storage might be a SAN-attached device, Fibre channel attached device or TCP/IP attached device. The device might be attached to all nodes simultaneously, or the cluster resource manager can attach it to the active node only. The device might, in turn, be in turn a resilient one, presenting the distributed file system with software or hardware replication between filesystem nodes. In the case of a geographically distributed cluster, the shared storage also can be distributed geographically in the same way as cluster nodes, and the cluster resource manager can attach the storage instance to the node the storage instance in the same geo-location.

Two or three nodes constitute a cluster and at any moment only one of them runs can be running on FIXEdge. When a failure of the active FIXEdge node is detected, the shared FIX message file system storage is unmounted on that node and mounted on the second node. Then, then FE is started on the second node. All sessions are handled by one active server and will be started on another node in case of failure.

FIXEdge start-up time grows increases with the number of the session it serves. 

Info

The current approach prevents load balancing between the cluster nodes.

The problem with balancing the load is resolved in this solution: FIXEdge NGE 

...

  • The simplest is to use a shared network filesystem (e.g. NFS share)
    The drawback is significantly increased latency (approx 20 times slower).
  • The recommended configuration is to use a SAN storage that can be attached to every each of the cluster nodes.
    The storage itself is a block device that is mounted on the node where the FIXEdge is currently runrunning.
    This approach allows having shared storage for all nodes while keeping I/O latency low.

...

  • Start FIXEdge at start-of-day - via pacemaker per FE resource by enabling resources (fixegde).
  • Stop FIXEdge at end-of-day, for maintenance hours - via pacemaker by disabling resources (fixegde).
  • Log archiving - mandatory for FIXEdge operation, FIXEdge needs available space on on the /opt/fixedge/FIXEdge1/log directory, so a periodic archiving of the files and directory cleanup is required.

...

  1. Install packages from a repository:

    Code Block
    languagebash
    yum install corosync pcs pacemaker 
  2. Set the password for the hacluster  user:

    Code Block
    languagebash
    passwd hacluster 
  3. Open ports on the firewall:

    Code Block
    languagebash
    firewall-cmd --add-service=high-availability 
    firewall-cmd --runtime-to-permanent 
  4. Enable cluster services to run at the system start-up:

    Code Block
    languagebash
    systemctl enable pcsd corosync pacemaker 

...

To install FIXEdge you first need the following artifacts:

  • the FIXEdge package (fixedge.tar.gz)
  • FIXEdge systemd  integration configuration (fixedge-systemd.tar.gz)
  • the license (engine.license). 

Superuser privileges are required for all the steps.

  1. Install the FIXEdge RPM package:

    Code Block
    languagebash
    mkdir --parents /opt/fixedge
    tar --extract --file fixedge.tar.gz --directory /opt/fixedge 
  2. Unpack the FIXEdge systemd integration configuration:

    Code Block
    languagebash
    tar --extract --file fixedge-systemd.tar.gz --directory /
  3. Add a user and a group for the FIXEdge:

    Code Block
    languagebash
    groupadd --system fixedge 
    useradd --system --gid fixedge --home-dir /opt/fixedge --shell /sbin/nologin --comment "Account to own and run FIXEdge" fixedge 
  4. Change ownership of the FIXEdge to the dedicated user:

    Code Block
    languagebash
    chown --recursive fixedge:fixedge /opt/fixedge
  5. Copy the license:

    Code Block
    languagebash
    cp engine.license /opt/fixedge 
  6. Enable the FIXICC Agent to start at the system start-up:

    Code Block
    systemctl enable fixicc-agent 

...

Prepare Storage for the Session Logs and the Configuration

At this point, FIXEdge is deployed locally. Now, now we need to make the configuration and the state shared.

...

  1. Create a filesystem on the logical volume:

    Code Block
    languagebash
    mkfs -t xfs /dev/shared_logs_group/shared_logs 
  2. Mount the filesystem:

    Code Block
    languagebash
    mount /dev/shared_logs_group/shared_logs /opt/fixedge/FIXEdge1/log 
  3. Create a FIXEdge log directory structure:

    Code Block
    languagebash
    mkdir /opt/fixedge/FIXEdge1/log/archive /opt/fixedge/FIXEdge1/log/backup 
  4. Unmount and deactivate:

    Code Block
    languagebash
    umount /opt/fixedge/FIXEdge1/log
    vgchange -a n shared_logs_group

...

Now we need to set up cluster resources for the FIXEdge. Superuser privileges are required for all the steps.

  1. Authorize nodes for hacluster  user:

    Code Block
    languagebash
    pcs cluster auth NODE_1_NAME NODE_2_NAME -u hacluster 

    Where NODE_1_NAME  and NODE_2_NAME are the hostnames of the servers which that run FIXEdge.

    Info
    Note that these names must not resolve to 127.0.0.1 locally or use IP addresses instead of the hostnames.
  2. Create the cluster and add nodes:

    Code Block
    languagebash
    pcs cluster setup --force --name fixedge_ha NODE_1_NAME NODE_2_NAME 

    Where NODE_1_NAME and NODE_2_NAME are the hostnames of the servers which that run FIXEdge.

    Info
    Note that these names must not resolve to 127.0.0.1 locally or use IP addresses instead of the hostnames.
  3. Start cluster

    Code Block
    languagebash
    pcs cluster start --all 

    Now the cluster is starting up, you can check its status with commands:

    Code Block
    languagebash
    pcs status cluster
    pcs status nodes
  4. Disable resource migration on the first failure, since restarting on the same node takes less time than migration to another node:

    Code Block
    languagebash
    pcs property set start-failure-is-fatal=false 
  5. For a two-nodes node cluster we must disable the quorum, but do not do this for a three-node cluster. Quorum avoids the situation when the cluster cannot decide which node is active.

    Code Block
    languagebash
    pcs property set no-quorum-policy=ignore 
  6. Add a virtual IP as a resource to the cluster:

    Code Block
    languagebash
    pcs resource create virtual_ip ocf:heartbeat:IPaddr2 ip=<VIRTUAL_IP> cidr_netmask=32 op monitor interval=30s 

    where <VIRTUAL_IP> is the IP that will be used by the FIX clients to connect to the cluster.

Use a Dual-Port SAN Device
  1. Add an LVM group for the sessions' logs as a resource to the cluster:

    Code Block
    languagebash
    pcs resource create logs_vg ocf:heartbeat:LVM volgrpname=logs 
  2. Add a filesystem for the sessions' logs as a resource to the cluster:

    Code Block
    languagebash
    pcs resource create logs_fs ocf:heartbeat:Filesystem device=/dev/shared_logs_group/shared_logs directory=/opt/fixedge/FIXEdge1/log fstype=xfs 
  3. Add FIXEdge as a resource to the cluster:

    Code Block
    languagebash
    pcs resource create fixedge systemd:fixedge op start timeout=300s op stop timeout=60s op monitor interval=10 timeout=60s meta migration-threshold=3 
  4. Make sure that all resources are started on the same node:

    Code Block
    languagebash
    pcs constraint colocation set virtual_ip logs_vg logs_fs fixedge sequential=true setoptions score=INFINITY 
  5. Make sure that resources are started in the proper order:

    Code Block
    languagebash
    pcs constraint order set virtual_ip logs_vg logs_fs fixedge 

...

Use FIXICC to connect to the cluster nodes: add servers with addresses of the cluster nodes. The FIXICC will show one server running and another stopped.

...

Open https://NODE_1_NAME:2224/ to access the pacemaker's WebUI. It provides means for inspecting the state of the cluster nodes and resources. Under normal operation, all nodes and resources must be running. All resources must be running the same node.

...

In this setup, the TCP/IP attached device is used. A two nodes -node GlusterFS file server with mirroring is used as the shared device. The device is mounted to both nodes simultaneously. See the diagram below:

...

  1. Download and install:

    Code Block
    languagebash
    $ sudo wget -P /etc/yum.repos.d http://download.gluster.org/pub/gluster/glusterfs/LATEST/CentOS/glusterfs-epel.repo
    $ sudo yum install glusterfs
    $ sudo yum install glusterfs-fuse
    $ sudo yum install glusterfs-server
  2. Check installed version:

    Code Block
    languagebash
    $ glusterfsd --version
    
    glusterfs 3.6.2 built on Jan 22 2015 12:58:10
    Repository revision: git://git.gluster.com/glusterfs.git
    Copyright (c) 2006-2013 Red Hat, Inc. <http://www.redhat.com/>
    GlusterFS comes with ABSOLUTELY NO WARRANTY.
    It is licensed to you under your choice of the GNU Lesser
    General Public License, version 3 or any later version (LGPLv3
    or later), or the GNU General Public License, version 2 (GPLv2),
    in all cases as published by the Free Software Foundation.
    
  3. Start glusterfs services on all servers with and enable them to start automatically on startup:

    Code Block
    languagebash
    $ sudo /etc/init.d/glusterd start
    $ sudo chkconfig glusterfsd on

...

Setting up FIXEdge instances

Copy FixEdgeCopy the FixEdge-*.tar.gz Linux package to /home/user to both nodes: NODE_1_NAME and NODE_2_NAME

...

  1. Unpack the FIXEdge-x.x.x.tar.gz archive to /home/user/

    Code Block
    languagebash
    $ tar xf FixEdge-5.8.2.68334-Linux-2.6.32-gcc447-x86_64.tar -C /home/user/

    The installation directory will be /home/user/FIXEdge:

  2. Copy the license 'engine.license to ' to the /home/user/FIXEdge folder.
  3. Move FIXEdge configuration and logs directory to shared storage mounted to /data:

    Code Block
    languagebash
    mv FIXEdge1 /data/
  4. Edit scripts in /home/user/FIXEdge/bin to direct them to the new FIXEdge instance location:
    • replace ".." with "/data".
  5. Edit FIXICC Agent configuration:
    • /home/user/FIXEdge/fixicc-agent/conf/wrapper.conf
      • wrapper.daemon.pid.dir = ${wrapper_home}
    • /home/user/FixEdge/fixicc-agent/conf/agent.properties 
      • EngineProperty = /data/FIXEdge1/conf/engine.properties
      • FIXEdgeFileSettings = /data/FIXEdge1/conf/FIXEdge.properties
      • LogUrl = /data/FIXEdge1/log
  6. Edit FIXEdge and engine configuration:
    • /data/FixEdge1/conf/engine.properties 
      • EngineRoot = /data
      • LicenseFile = /home/user/FIXEdge/engine.license
    • /data/FixEdge1/conf/FIXEdge.properties
      • FIXEdge.RootDir = /data/FIXEdge1
      • Log.File.RootDir = /data/FIXEdge1
      • TransportLayer.SmtpTA.DllName = /home/user/FIXEdge/bin/libSMTPTA.so
  7. Install and Start FIXICC Agent daemon:

    Code Block
    languagebash
    $ cd /home/user/FIXEdge/fixicc-agent/bin
    $ ./installDaemon.sh
    $ ./startDaemon.sh
  8. Now everything is ready to run FIXEdge on Node 1.
  9. Prepare to copy the installation to Node 2:

    Code Block
    languagebash
    $ cd /home/user
    $ tar cvf FixEdge.tar FixEdge
    $ gzip FixEdge.tar
  10. Copy file FixEdge.tar.gz to Node2:/user/home

...

  1. Unpack the FIXEdge-x.x.x.tar.gz archive to /home/user/

    Code Block
    languagebash
    $ tar xf FixEdge-5.8.2.68334-Linux-2.6.32-gcc447-x86_64.tar -C /home/user/
  2. Install and Start the FIXICC Agent daemon:

    Code Block
    languagebash
    $ cd /home/user/FIXEdge/fixicc-agent/bin
    $ ./installDaemon.sh
    $ ./startDaemon.sh

...

https://github.com/feist/pcs/issues/

  1. On both nodes, install the needed software:

    Code Block
    languagebash
    $ sudo yum install corosync pcs pacemaker 
  2. On both nodes, set the password for hacluster user ('epmc-cmcc' was used):

    Code Block
    languagebash
    $ sudo passwd hacluster
  3. Configure Firewall on both nodes to allow cluster traffic:

    Code Block
    languagebash
    $ sudo iptables -I INPUT -m state --state NEW -p udp -m multiport --dports 5404,5405 -j ACCEPT
    $ sudo iptables -I INPUT -p tcp -m state --state NEW -m tcp --dport 2224 -j ACCEPT
    $ sudo iptables -I INPUT -p igmp -j ACCEPT
    $ sudo iptables -I INPUT -m addrtype --dst-type MULTICAST -j ACCEPT
    $ sudo service iptables save
  4. Start the pcsd service on both nodes:

    Code Block
    languagebash
    $ sudo systemctl start pcsd
  5. From now on, all commands needs need to be executed on one node only. We can control the cluster by using PCS from on one of the nodes.
    Since we will configure all nodes from one point, we need to authenticate on all nodes before we are allowed to change the configuration. Use the previously configured hacluster user and password to do this:

    Code Block
    languagebash
    $ sudo pcs cluster auth NODE_1_NAME NODE_2_NAME 
    Username: hacluster
    Password:
    NODE_1_NAME: Authorized
    NODE_2_NAME: Authorized
  6. Create the cluster and add nodes. This command creates the cluster node configuration in /etc/corosync.conf.

    Code Block
    languagebash
    $ sudo pcs cluster setup --name fixedge_cluster NODE_1_NAME NODE_2_NAME 
    Shutting down pacemaker/corosync services...
    Redirecting to /bin/systemctl stop  pacemaker.service
    Redirecting to /bin/systemctl stop  corosync.service
    Killing any remaining services...
    Removing all cluster configuration files...
    NODE_1_NAME: Succeeded
    NODE_2_NAME: Succeeded
  7. We can start cluster now:

    Code Block
    languagebash
    $ sudo pcs cluster start --all
    NODE_1_NAME: Starting Cluster...
    NODE_2_NAME: Starting Cluster...
  8. We can check cluster status:

    Code Block
    languagebash
    $ sudo pcs status cluster
    Cluster Status:
     Last updated: Tue Jan 27 22:11:15 2015
     Last change: Tue Jan 27 22:10:48 2015 via crmd on NODE_1_NAME
     Stack: corosync
     Current DC: NODE_1_NAME (1) - partition with quorum
     Version: 1.1.10-32.el7_0.1-368c726
     2 Nodes configured
     0 Resources configured
    
    
    $ sudo pcs status nodes
    Pacemaker Nodes:
     Online: NODE_1_NAME NODE_2_NAME
     Standby:
     Offline:
     
    $ sudo corosync-cmapctl | grep members
    runtime.totem.pg.mrp.srp.members.1.config_version (u64) = 0
    runtime.totem.pg.mrp.srp.members.1.ip (str) = r(0) ip(10.17.131.127)
    runtime.totem.pg.mrp.srp.members.1.join_count (u32) = 1
    runtime.totem.pg.mrp.srp.members.1.status (str) = joined
    runtime.totem.pg.mrp.srp.members.2.config_version (u64) = 0
    runtime.totem.pg.mrp.srp.members.2.ip (str) = r(0) ip(10.17.131.128)
    runtime.totem.pg.mrp.srp.members.2.join_count (u32) = 1
    runtime.totem.pg.mrp.srp.members.2.status (str) = joined
    
    
    $ sudo pcs status corosync
    Membership information
    ----------------------
        Nodeid      Votes Name
             1          1 NODE_1_NAME (local)
             2          1 NODE_2_NAME
  9. Disable the STONITH option as we don't have STONITH devices in our demo virtual environment:

    Code Block
    languagebash
    $ sudo pcs property set stonith-enabled=false
  10. For a two-nodes node cluster we must disable the quorum:

    Code Block
    languagebash
    $ sudo pcs property set no-quorum-policy=ignore
    $ sudo pcs property
    Cluster Properties:
     cluster-infrastructure: corosync
     dc-version: 1.1.10-32.el7_0.1-368c726
     no-quorum-policy: ignore
     stonith-enabled: false
  11. Add Virtual IP as a resource to the cluster:

    Code Block
    languagebash
    $ sudo pcs resource create virtual_ip ocf:heartbeat:IPaddr2 ip=10.17.135.17 cidr_netmask=32 op monitor interval=30s
    $ sudo pcs status resources
     virtual_ip (ocf::heartbeat:IPaddr2): Started
  12. Add FIXEdge as a resource to cluster:

    Code Block
    languagebash
    $ sudo pcs resource create FIXEdge ocf:heartbeat:anything params binfile="/home/user/FixEdge/bin/FIXEdge" cmdline_options="/data/FixEdge1/conf/FIXEdge.properties" user="user" logfile="/home/user/FIXEdge_resource.log" errlogfile="/home/user/FIXEdge_resource_error.log"
    Note

    For some reason in the /usr/lib/ocf/resource.d/ of the installed cluster there are many missing agents, including ocf:heartbeat:anything. You need to modify the original version (which you can download here: https://github.com/ClusterLabs/resource-agents/blob/master/heartbeat/anything) to make it working. The working version of the agent is attached.

    This file should be copied to /usr/lib/ocf/resource.d/ and make executable:

    Code Block
    languagebash
    $ sudo cp anything /usr/lib/ocf/resource.d/heartbeat/
    $ sudo chmod a+rwx /usr/lib/ocf/resource.d/heartbeat/anything  

    Also, to make this agent works the following lines shall be added to sudoers file:

    Code Block
    languagebash
    $ sudo visudo
    Defaults    !requiretty
    user    ALL=(user)      NOPASSWD: ALL
    root    ALL=(user)      NOPASSWD: ALL
  13. In order to make sure that the Virtual IP and FIXEdge always stay together, we can add a constraint:

    Code Block
    languagebash
    $ sudo pcs constraint colocation add FIXEdge virtual_ip INFINITY
  14. To avoid the situation where the FIXEdge would start before the virtual IP is started or owned by a certain node, we need to add another constraint which that determines the order of availability of both resources:

    Code Block
    languagebash
    $ sudo pcs constraint order virtual_ip then FIXEdge
    Adding virtual_ip FIXEdge (kind: Mandatory) (Options: first-action=start then-action=start)
  15. After configuring the cluster with the correct constraints, restart it and check the status:

    Code Block
    languagebash
    $ sudo pcs cluster stop --all && sudo pcs cluster start --all
    NODE_1_NAME: Stopping Cluster...
    NODE_2_NAME: Stopping Cluster...
    NODE_2_NAME: Starting Cluster...
    NODE_1_NAME: Starting Cluster...
  16. The cluster configuration is now completed.

...

File

Size

Created

Comment

587268Mar 06, 2015 21:49Modified Sender source code (Sending orders with incremented ClOrdID)
9475Jan 28, 2015 09:16ocf:heartbeat:anything agent for FIXEdge resource monitoring
4976201Mar 08, 2015 01:06Clients simulator (binary)
34 KBJan 20, 2020

FIX Logs Replicator

The FIX Antenna Replication tool (RT) is typically used for FIX session real-time replication on a backup host with FIX engine.

...

Failure of software or hardware within a node (e.g. application failure, node unavailability, resource exhausting). Failover Cluster consists of several application nodes. In case of one node fails , the application (or DB) is moved to another node and starts there within approximately 2 minutes.

...

Achievable RPO Values: "Recent synch point: Greater than 0 seconds, up to and including 1 day, i.e. since the start of the day". 

Info

The better Better RPO values can be achieved with additional configuration or tools like the log replication tool.

This procedure requires additional configuration for each client individually.

...