Starting troubleshooting from “where the loss occurs” is more effective than discussing `acks` in general terms. It is recommended to locate issues by segmenting the pipeline:
1. Producer side: First verify whether it is a “false success”
When `acks=0`, a successful `send()` response does not mean the broker has received the message. With `acks=1`, the response is returned once the leader writes the record. If the leader crashes before replicating to followers, data loss happens. For guaranteed no loss, `acks=all` plus `min.insync.replicas>=2` and `replication.factor>=3` are all required. Changing only `acks` is a common pitfall.
Conflict between retries and ordering: If retries are enabled without idempotence, retries cause duplicates instead of loss. However, if `delivery.timeout.ms` is set too low (smaller than `request.timeout` plus `linger`), retries get terminated early and result in apparent loss. Ensure `delivery.timeout.ms >= linger.ms + request.timeout.ms`.
An idempotent producer (`enable.idempotence=true`) guarantees no duplicates or loss only within a single partition and a single session, not across sessions. Changes to broker-side `transaction.max.timeout.ms` or PID rotation after producer restarts can still lead to duplicates. Exactly-once semantics across sessions require transactions with `transactional.id`.
2. Broker side: Focus on out-of-sync replicas being elected as leader
`unclean.leader.election.enable=true` is a hidden source of data loss. Promoting an OSR replica truncates committed records, so this should be set to `false` in production.
Avoid misconceptions about `log.flush` settings: Kafka relies on replicas rather than fsync for durability. Setting `flush.messages=1` only degrades performance and does not prevent loss.
3. Consumer side: This is where most perceived “loss” actually takes place
With `enable.auto.commit=true` and asynchronous or multi-threaded processing, offsets get committed before message processing completes. If the process crashes, messages are lost. Switch to manual commit and commit only after successful processing.
When using `seek` for backtracking or `auto.offset.reset=latest` for new consumer groups, records in the gap may be skipped. Check the gap between consumer group offsets and the log end offset during investigation.
Troubleshooting workflow: First run `kafka-consumer-groups.sh –describe` to check if lag resets unexpectedly. Next inspect broker logs for `Shrinking ISR` or `LeaderElection` events. Finally add producer callbacks to collect `RecordMetadata.offset` and exception metrics, to confirm whether the issue stems from send failures or skipped consumption. It is recommended to implement reconciliation checkpoints with unique message IDs at both ends of the pipeline for faster root-cause analysis instead of guessing configurations after incidents.