Page tree
Skip to end of metadata
Go to start of metadata

The Kafka endpoint is a FEJ transport adapter, which provides connectivity to the Kafka streaming platform. Its implementation is based on the FIX Antenna Java Kafka adapter and it uses Kafka Producer and Consumer API internally.

Kafka endpoint parameters are described in the conf/kafka-adaptor.properties file. The configuration should include a list of Kafka endpoints and link them to the corresponding Kafka topics.

Configuration properties

The Kafka endpoints are configured by means of the following properties:

Property Name

Description

RequiredDefault Value
kafka.clientsComma-delimited list of Kafka endpoints.
A separate configuration section for each listed client should be specified
Y

Generic Kafka parameters

These parameters are applied to all producer and consumer endpoints

kafka.[kafka_properties]

This section may include multiple options, which are applied to all producer and consumer endpoints. As a [kafka_property] may be used any key from Kafka Producer Configs or Kafka Consumer Config.

For example, it's a good section to define a bootstrap server for all endpoints:

kafka.bootstrap.servers=localhost:9092

N

Generic Kafka Producer parameters

kafka.producer.key.serializer

The serializer class for a record key that implements the com.epam.fej.kafka.KafkaEndpointKeySerializer interface.N

kafka.producer.value.serializer

The serializer class for a record value that implements the com.epam.fej.kafka.KafkaEndpointValueSerializer interface.N

kafka.producer.[kafka_properties]

This section may include multiple options, which are applied to all producer endpoints. As a [kafka_property] may be used any key from Kafka Producer Configs.

N

Kafka Producer parameters

kafka.producer.[ClientName].client.id

The ID string to pass to the server when making requests. The purpose of this is to be able to track the source of requests beyond just IP/port by allowing a logical application name to be included in server-side request logging.

Y

kafka.producer.[ClientName].topic

The topic for publishing by the given Kafka producer endpoint.Y

kafka.producer.[ClientName].async.sending

Defines the sending mode for this producer endpoint.

If the value is true, the endpoint sends data asynchronously.

If the value is false, it waits for confirmation from the Kafka broker for every sent message.

Nfalse

kafka.producer.[ClientName].key.serializer

The serializer class for a record key that implements the com.epam.fej.kafka.KafkaEndpointKeySerializer interface.Y, if the kafka.producer.key.serializer property is not defined

kafka.producer.[ClientName].value.serializer

The serializer class for a record value that implements the com.epam.fej.kafka.KafkaEndpointValueSerializer interface.Y, if the kafka.producer.value.serializer property is not defined

kafka.producer.[ClientName].groups

The list of groups for routingN

kafka.producer.[ClientName].startOnload

Specifies whether a producer should be started during the FIXEdge Java server initialization.Ntrue

kafka.producer.[ClientName].startTime

A cron expression that defines a producer's start time.N

kafka.producer.[ClientName].stopTime

A cron expression that defines a producer's stop time.N

kafka.producer.[ClientName].scheduleTimeZone

A time zone for the start and stop timesN

kafka.producer.[ClientName].[kafka_properties]

This section may include multiple options, which are applied to all producer endpoints. As a [kafka_property] may be used any key from Kafka Producer Configs.N

Generic Kafka Consumer parameters

kafka.consumer.key.deserializer

The deserializer class for a record key that implements the com.epam.fej.kafka.KafkaEndpointKeyDeserializer interface.N

kafka.consumer.value.deserializer

The deserializer class for a record value that implements the com.epam.fej.kafka.KafkaEndpointValueDeserializer interface.N

kafka.consumer.[kafka_properties]

This section may include multiple options, which are applied to all producer endpoints. As a [kafka_property] may be used any key from Kafka Consumer Configs.

N

Kafka Consumer parameters

kafka.consumer.[ClientName].client.id

The ID string to pass to the server when making requests. The purpose of this is to be able to track the source of requests beyond just IP/port by allowing a logical application name to be included in server-side request logging.Y

kafka.consumer.[ClientName].topics

The list of topics to subscribe toY, if the kafka.consumer.[ClientName].topics.regexp property is not defined

kafka.consumer.[ClientName].topics.regexp

The pattern to subscribe toY, if the  kafka.consumer.[ClientName].topics property is not defined

kafka.consumer.[ClientName]

The time, in milliseconds, which consumer spends waiting in the polling procedure if data is not available in the buffer. If 0, returns immediately with any records that are available currently in the buffer, else returns empty. Must not be negative.N200

kafka.consumer.[ClientName].key.deserializer

The deserializer class for a record key that implements the com.epam.fej.kafka.KafkaEndpointKeyDeserializer interface.Y, if the kafka.consumer.key.deserializer property is not defined

kafka.consumer.[ClientName].value.deserializer

The deserializer class for a record value that implements the com.epam.fej.kafka.KafkaEndpointValueDeserializer interface.Y, if the kafka.consumer.value.deserializer property is not defined

kafka.consumer.[ClientName].groups

The list of groups for routingN

kafka.consumer.[ClientName].startOnload

Specifies whether a consumer should be started during FIXEdge Java server initialization.Ntrue

kafka.consumer.[ClientName].startTime

A cron expression that defines a consumer's start time.N

kafka.consumer.[ClientName].stopTime

A cron expression that defines a consumer's stop time.N

kafka.consumer.[ClientName].scheduleTimeZone

A time zone for the start and stop timesN

kafka.consumer.[ClientName].[kafka_properties]

This section may include multiple options, which are applied to all producer endpoints. As a [kafka_property] may be used any key from Kafka Consumer Configs.N

For the full list of Kafka properties refer to the Kafka official documentation.

NOTE: All defined Kafka endpoints are started automatically during FIXEdge Java server initialization by default if there is a defined schedule for them (starting from FEJ 1.8.0). The schedule may be defined by startTime/stopTime properties or within a separate schedule configuration. Refer to the Scheduler section for more details.

NOTE: All changes in the properties file are applied only after the FIXEdge Java server restart.

Sample configuration

The sample of the simplest config with Kafka producer and consumer endpoints:

# list of kafka endpoints (consumers and producer)
kafka.clients = KProducer, KConsumer

# main properties to specify producer's id and topic
kafka.producer.KProducer.client.id = KProducer
kafka.producer.KProducer.topic = PTopic1
# Type is class. Serializer class for key that implements the com.epam.fej.kafka.KafkaEndpointKeySerializer interface.
kafka.producer.key.serializer = com.epam.fej.kafka.FIXMessageEventSerializer
# Type is class. Serializer class for value that implements the com.epam.fej.kafka.KafkaEndpointValueSerializer interface.
kafka.producer.value.serializer = com.epam.fej.kafka.FIXMessageEventSerializer


# main properties to specify consumer's id and topic
kafka.consumer.KConsumer.client.id = KConsumer
kafka.consumer.KConsumer.topics = KTopic2
# Type is class. Deserializer class for key that implements the com.epam.fej.kafka.KafkaEndpointKeyDeserializer interface.
kafka.consumer.key.deserializer = org.apache.kafka.common.serialization.StringDeserializer
# Type is class. Deserializer class for value that implements the com.epam.fej.kafka.KafkaEndpointValueDeserializer interface.
kafka.consumer.value.deserializer = com.epam.fej.kafka.FIXMessageEventDeserializer
  • No labels