Page tree

Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

When several clients (either one Producer or one Consumer) are configured for the Kafka TA, all of them establish connection to the Kafka platform when FIXEdge starts its work.
If one of the clients fails to establish connection, the remaining clients are not affected by this failure.

Delivery

To be defined

The Kafka TA provides guaranteed message delivery.

Async message handling

When a message arrives at the particular Kafka topic in the defined order, the messages are received by FIXEdge via Consumer API and the messages are available at the FIXEdge business layer in the same order.

When the FIXEdge BL reports to the Kafka TA that the message is delivered and the "Commit" parameter is set to "Sync" for Consumer, and FIX messages to Kafka via Producer API are sent in the same order in which FIXEdge sends the messages to the Client. FIXEdge receives (?) FIX messages via Consumer API in the same order in which messages arrive at the particular Kafka topic.

When the "Commit" parameter is not configure or set to "Auto" for Consumer, and Kafka sends the message to FIXEdge, then the Kafka TA sends the commit to Kafka .

Configuration steps

To set up the Kafka TA for FIXEdge, the following steps have to be executed:

...

Enable a transport adapter to be used by FIXEdge:
In the ‘Transport Layer Section’ of the FIXEdge.properties, add the Kafka TA to the list of supported adapters:

Code Block
languagetext
TransportLayer.TransportAdapters = TransportLayer.KafkaTA

Note: If you use other transport adapters, just add TransportLayer.KafkaTA to the end of the list:

Code Block
languagetext
TransportLayer.TransportAdapters = TransportLayer.SmtpTA, TransportLayer.RestTA, TransportLayer.KafkaTA

Configure the Kafka TA by adding the Kafka TA section to the FIXEdge.properties:

...

languagetext

...

when the configured time interval expires.

When the "GroupId" parameter is not configured and its value is to be assigned, the session ID is used as a value of the Kafka TA "GroupId" parameter.

Async message handling

When a message arrives at the particular Kafka topic in the defined order, the messages are received by FIXEdge via Consumer API and the messages are available at the FIXEdge Business Layer in the same order.

When the FIXEdge BL reports to the Kafka TA that the message is delivered and the "Commit" parameter is set to "Sync" for Consumer, and Kafka sends the message to FIXEdge, then the Kafka TA sends the commit to Kafka.

Configuration steps

To set up the Kafka TA for FIXEdge, the following steps have to be executed:

  1. Copy the FIXEdge distribution package to the FIX Edge home:
    Unpack the content of the distribution package to the FIXEdge root directory, e.g. to C:\FIXEdge

  2. Enable a transport adapter to be used by FIXEdge:
    In the ‘Transport Layer Section’ of the FIXEdge.properties, add the Kafka TA to the list of supported adapters:

    Code Block
    languagetext
    TransportLayer.TransportAdapters = TransportLayer.KafkaTA

    Note: If you use other transport adapters, just add TransportLayer.KafkaTA to the end of the list:

    Code Block
    languagetext
    TransportLayer.TransportAdapters = TransportLayer.SmtpTA, TransportLayer.RestTA, TransportLayer.KafkaTA
  3. Configure the Kafka TA by adding the Kafka TA section to the FIXEdge.properties:

    Code Block
    languagetext
    TransportLayer.KafkaTA.Description = Kafka Transport Adaptor
    TransportLayer.KafkaTA.DllName = bin/KafkaTA-vc10-MD-x64.dll
    TransportLayer.KafkaTA.Sessions = Kafka
     
    TransportLayer.KafkaTA.Kafka.bootstrap.servers = localhost:9092
    TransportLayer.KafkaTA.Kafka.FIXVersion = FIX44
    
    TransportLayer.KafkaTA.Kafka.Consumer.Commit = Auto
    TransportLayer.KafkaTA.Kafka.Consumer.Topics = outputTopic
    TransportLayer.KafkaTA.Kafka.Consumer.group.id = ID
    
    TransportLayer.KafkaTA.Kafka.Producer.Topic = topic

    Note: Sample settings can be copied to the FIXEdge.properties file from the KafkaTA.properties file (located in the doc folder of the FIXEdge distribution package).

  4. Configure rules for message routing from the Kafka TA.
    The KafkaTA Client is referred to the business layer Business Layer (BL) by ClientID name specified in the FIXEdge.properties file.  For Kafka Business Layer configuration sample, refer to the Configuration sample sub-section.

  5. Restart the FIXEdge server to apply the changes.

...

Code Block
languagexml
titleBL_Config
collapsetrue
<FIXEdge>
    <BusinessLayer>

       <Rule>
            <Source>
                <FixSession SenderCompID="SC" TargetCompID="FE"/>
            </Source>
            <Action>
                <Send><Client Name="Kafka"/></Send>
            </Action>
        </Rule>
       <Rule>
            <Source>
                <Client Name="Kafka"/>
            </Source>
            <Action>
                <Send>
                    <FixSession SenderCompID="FE" TargetCompID="SC"/>
                </Send>
            </Action>
        </Rule>

        <DefaultRule>
            <Action>
                <DoNothing/>
            </Action>
        </DefaultRule>

    </BusinessLayer>
</FIXEdge>


Logging

Any configuration action is logged Logging messages differ by type:

  • INFO
  • DEBUG
  • WARN
  • ERROR

Any configuration action is logged in the "Kafka_TA" log category in the log file via the INFO messages. All the config parameters applied as a result of configuration are logged.

...

When when FIX messages are transferred in the raw format (i.e. serialization is not applied), the Kafka TA implies a string containing the tag values converted to strings. 

The serialized format of communication is fully supported by the Kafka TA. Custom serialization implies the specially designed external code and includes serialization of all Kafka message parts:

  • Key
  • Value
  • Header(-s)

Serialization on sending

When custom serialization is applied and FIXEdge sends the message to the particular Kafka topic, this message is serialized via the custom serializer and is sent to the particular Kafka topic via Producer API. The following INFO message is logged in the client log category of the log file:

"The message content:
headers: <headers>,
key: <key>,
partition Id: <partition Id>,
value: <value>" 

Serialization on receiving

When custom serialization is applied and the new message arrives at the particular Kafka topic, this message is received by FIXEdge via Consumer API. The following DEBUG message is logged in the client log category of the log file:

"The message content:
headers: <headers>,
key: <key>,
partition Id: <partition Id>,
value: <value>" 

The message is deserialized via the custom serializer.

Message key processing

In case of the default serialization, the particular tag is configured in the keytag parameter and the specified key is passed to the Kafka platform along with the message sent to the particular Kafka topic.

In case of the custom serialization, the custom plugin produces the key which is passed to the Kafka platform along with the message sent to the particular Kafka topic.

Custom partitioning

If the custom partitioner is configured, the specified key and/or topic should be passed to the FIXEdge business layer via the configured tag of a FIX messageof communication is fully supported by the Kafka TA. Custom serialization implies the specially designed external code and includes serialization of all Kafka message parts:

  • Key
  • Value
  • Header(-s)

Serialization on sending

When custom serialization is applied and FIXEdge sends the message to the particular Kafka topic, this message is serialized via the custom serializer and is sent to the particular Kafka topic via Producer API. The following INFO message is logged in the client log category of the log file:

"The message content:
headers: <headers>,
key: <key>,
partition Id: <partition Id>,
value: <value>" 

Serialization on receiving

When custom serialization is applied and the new message arrives at the particular Kafka topic, this message is received by FIXEdge via Consumer API. The following DEBUG message is logged in the client log category of the log file:

"The message content:
headers: <headers>,
key: <key>,
partition Id: <partition Id>,
value: <value>" 

The message is deserialized via the custom serializer.

Message key processing

In case of the default serialization, the particular tag is configured in the keytag parameter and the specified key is passed to the Kafka platform along with the message sent to the particular Kafka topic.

In case of the custom serialization, the custom plugin produces the key which is passed to the Kafka platform along with the message sent to the particular Kafka topic.

Custom partitioning

If the custom partitioner is configured, the specified key and/or topic should be passed to the FIXEdge Business Layer via the configured tag of a FIX message.


Message Content Wrapping

When BL Client is configured to work with XMLData (213), and FIXEdge sends a message to the particular Kafka topic, and the XMLData (213) field exists in the FIX message and this field is not empty, then the value of the XMLData (213) field is sent to the particular Kafka topic via Producer API.

When BL Client is configured to work with XMLData (213), and FIXEdge sends a message to the particular Kafka topic, and the XMLData (213) field does not exist in the FIX message and this field is empty, then nothing is sent to the particular Kafka topic via Producer API. The corresponding error is sent to the Business Layer.

When BL Client is configured to work with XMLData (213), and Kafka sends a message to FIXEdge, then the FIX message with 35=n (XML message) is received by FIXEdge,
and the XMLData (213) field is filled in with the received message, and the XMLDataLen (212) field is filled in with the received message length.

Troubleshooting

Common troubleshooting issues include:

...

If the adapter is receiving messages from the Kafka server, but not delivering them to target sessions, Business Level Layer rules have either been improperly configured or have not been specified.

...

Expand
titleClick to see an example

With DEBUG logging enabled, we can see that message was received, but the message processing failed because no targets were configured in the BL. Business Level Layer rules for the 'Kafka2' session have not been specified.

No Format
2020-11-25 16:35:36,545 UTC DEBUG [Kafka_TA] 21964 Session 'Kafka2': Consumer: Receiving the message: 8=FIX.4.49=15235=D49=SC56=FE34=252=20201125-16:35:35.936212=4213=test11=BTC/USD_LimitB_GTC55=BTC/USD54=160=20160401-15:15:58.08038=0.1540=244=630.159=110=078
2020-11-25 16:35:36,545 UTC DEBUG [BL_RoutingTable] 21964 No BL rules found for message with ClientID 'Kafka2', executing DefaultRule.
2020-11-25 16:35:36,546 UTC DEBUG [CC_Layer] 21964 BL has processed a message. Number of client IDs for delivery :0. Number or FIX sessions for delivery :0.. Number or sources identifiers for delivery :0.

2020-11-25 16:35:36,546 UTC DEBUG [Kafka_TA] 21964 Session 'Kafka2': Consumer: Message processing failed