Apache Kafka stores the message with offset. The following are several ways the messages can be retrieved with time;
Message Creation
When the message is created append the runtime to the message in the producer class before publishing the message to the topic as follows:
String runtime = new Date.toString();
String msg = "Test Message " + runtime;
KeyedMessage<String,String> data = new KeyedMessage<String, String> (topic,msg);
producer.send(data);
The message can be retrieved using the runtime timestamp.
Apache Kafka System Tools
Apache Kafka provided following system tools:
- Consumer Offset Checker
- Dump Log Segment
- Export Zookeeper Offsets
- Get Offset Shell
- Import Zookeeper Offsets
- JMX Tool
- Kafka Migration Tool
- Mirror Maker
- Replay Log Producer
- Simple Consumer Shell
- State Change Log Merger
- Update Offsets In Zookeeper
- Verify Consumer Rebalance
The following function gives the time as a long value
Get Offset Shell
get offsets for a topic
|
required argument [broker-list], [topic]
Option Description
--broker-list <hostname:port,..., REQUIRED: The list of hostname and hostname:port> port of the server to connect to.
--max-wait-ms <Integer: ms> The max amount of time each fetch request waits. (default: 1000)
--offsets <Integer: count> number of offsets returned (default: 1)
--partitions <partition ids> comma separated list of partition ids. If not specified, will find offsets for all partitions (default)
--time <Long: timestamp in milliseconds / -1(latest) / -2 (earliest) timestamp; offsets will come before this timestamp, as in getOffsetsBefore >
--topic <topic> REQUIRED: The topic to get offsets from.
Kafka Low Level Consumer API
For message partition reading, the kafka.api.OffsetRequest class defines two constants: EarliestTime and LatestTime, to find the beginning of the data in the logs and the new messages stream.
Add Comment