Starting from FIXEdge v.5.12.0 there was added a business protection mechanism regulating incoming and outgoing traffic to optimize memory consumption and to prevent system from overloading. The mechanism includes monitoring of throughput and of number of incoming messages received during a session as well as outgoing queue monitoring for slow consumers detection.
- FIXEdge takes measurements to protect itself from higher throughput that may risk of system overloading. Such incidents (i.e. of throughput higher than expected) are monitored and reported. It is essential that the incoming message flow will not exceed substantially the maximum daily expected number of messages from each inbound connection. It is possible to set:
The maximum number of messages per second a FIX session can send to FIXEdge
The maximum number of messages per day a FIX session can send to FIXEdge
- FIXEdge protects itself from slow consumers by monitoring and reporting the sessions with larger outgoing message queue. FIXEdge provides monitoring and alerting functionality for the outbound queue in order to provide the support team with advanced notice of any possible threat to the system
In case either threshold is reached, the FIX session will be logged out manually in order to prevent further messages from being sent.
Management of Incoming queue
There is no an inbound queue in FIXEdge/FIX Antenna. A new message is read from TCP buffer after previous one is processed. In other words, a number of waiting messages are limited by TCP buffer size and controlled by the Operating System.
There were added the following parameters in FIXEdge.properties file defined on session level:
- IncomingMessagesLimit – a limit for messages received during a session. The session-level messages are counted as well as application level messages.
- IncomingThroughputLimit – a limit for incoming messages throughput.
Default value for both parameters is '0' - unlimited, no events will be created on business level in such case. Both parameters are available for registered and unregistered sessions.
Management of Outgoing queue
Slow consumers can cause excessive memory consumption, therefore there was added new session parameter in FIXEdge.properties file to control outgoing queue:
- OutgoingQueueSize – a limit for outgoing queue.
Default value for the parameter is '0' - unlimited, no events will be created on business level in such case. Parameter is available for registered and unregistered sessions.
An event OnNofificationEvent rises within FIXEdge BL each time when described thresholds are reached.
Real-time values of the parameters above can be got in a response to SessionStat request sent via admin-session to FIXEdge.
Slow Consumer Testing Example
Sessions configuration in FIXEdge.properties file:
Configure and enable SMTP TA in FIXEdge.properties file:
Setup routing rules and slow consumer emulation in BL_Config.xml:
Slow consumer delays are emulated by sleep.js:
Steps to reproduce
- Start FIXEdge with described configuration;
- Connect 'TEST' (TEST/FIXEdge) session to FIXEdge;
- Send bunch of 10 000 New Order Singles (35=D) messages with max speed to session 'TEST'
Session FIXEDGE/LOOP emulates slow consumer and process only 10 messages per second.
- When the limits are reached FIXEdge send emails via SMTP
Actual parameters values are defined for each system individually and may vary depending on system requirements and current system load.
The value of IncomingMessagesLimit = 10 000 from the example above is a very low limit for a real message processing and was used to demonstrate FIXEdge protection mechanism only. General recommendation for IncomingMessagesLimit is > 10000000 (10+ million) per session.
Recommended value for IncomingThroughputLimit is 10 000 msg/s per session. Higher throughput for a session can affect performance of other sessions.
OutgoingQueueSize more that 100 should warn of an abnormal situation. Normal outgoing queue size is ~ 0-5.