Bug

[KAFKA-1310] - Zookeeper timeout causes deadlock in Controller
[KAFKA-1311] - Add a flag to turn off delete topic until it is stable
[KAFKA-1315] - log.dirs property in KafkaServer intolerant of trailing slash
[KAFKA-1317] - KafkaServer 0.8.1 not responding to .shutdown() cleanly, possibly related to TopicDeletionManager or MetricsMeter state
[KAFKA-1323] - log.dirs server property no longer supports relative directories
[KAFKA-1350] - Fix excessive state change logging
[KAFKA-1355] - Reduce/optimize update metadata requests sent during leader election
[KAFKA-1358] - Broker throws exception when reconnecting to zookeeper
[KAFKA-1362] - Publish sources and javadoc jars
[KAFKA-1363] - testTopicConfigChangesDuringDeleteTopic hangs
[KAFKA-1365] - Second Manual preferred replica leader election command always fails
[KAFKA-1373] - LogCleaner assumes first dirty offset zero if there is no cleaner checkpoint
[KAFKA-1398] - Topic config changes can be lost and cause fatal exceptions on broker restarts

Task
[KAFKA-1413] - bump kafka version to 0.8.1.1 in gradle.properties

0.8.1 release

New Feature

[KAFKA-50] - kafka intra-cluster replication support
[KAFKA-188] - Support multiple data directories
[KAFKA-202] - Make the request processing in kafka asynchonous
[KAFKA-203] - Improve Kafka internal metrics
[KAFKA-235] - Add a 'log.file.age' configuration parameter to force rotation of log files after they've reached a certain age
[KAFKA-429] - Expose JMX operation to set logger level dynamically
[KAFKA-475] - Time based log segment rollout
[KAFKA-545] - Add a Performance Suite for the Log subsystem
[KAFKA-546] - Fix commit() in zk consumer for compressed messages
Improvement
[KAFKA-100] - ProducerShell should use high-level producer instead of SyncProducer
[KAFKA-133] - Publish kafka jar to a public maven repository
[KAFKA-139] - cross-compile multiple Scala versions and upgrade to SBT 0.12.1
[KAFKA-155] - Support graceful Decommissioning of Broker
[KAFKA-165] - Add helper script for zkCli.sh
[KAFKA-181] - Log errors for unrecognized config options
[KAFKA-187] - Add Snappy Compression as a Codec and refactor CompressionUtil and option on startup to select what the default codec
[KAFKA-193] - use by name parameter helper for logging and trait to include lazy logging and refactor code to use the new LogHelper
[KAFKA-246] - log configuration values used
[KAFKA-253] - Refactor the async producer to have only one queue instead of one queue per broker in a Kafka cluster
[KAFKA-258] - Remove broker.id from the broker.list config in the Producer
[KAFKA-267] - Enhance ProducerPerformance to generate unique random Long value for payload
[KAFKA-271] - Modify new FetchResponse object to remove the initial offset field
[KAFKA-281] - support multiple root log directories
[KAFKA-285] - Increase maximum value of log.retention.size
[KAFKA-314] - Go Client Multi-produce
[KAFKA-323] - Add the ability to use the async producer in the Log4j appender
[KAFKA-324] - enforce broker.id to be a non-negative integer
[KAFKA-325] - revisit broker config in 0.8
[KAFKA-349] - Create individual "Response" types for each kind of request and wrap them with "BoundedByteBufferSend", remove "xxResponseSend" types for all requests except "FetchRequest"
[KAFKA-365] - change copyright in NOTICE to 2012
[KAFKA-366] - add jmx beans in broker to track # bytes in consumer
[KAFKA-368] - use the pig core jar from maven instead of distributing it
[KAFKA-393] - Add constructor for message which takes both byte array offset and length
[KAFKA-408] - ProducerPerformance does not work with all producer config options
[KAFKA-437] - Unused var statement in ZookeeperConsumerConnectorTest
[KAFKA-439] - @returns was used in scala doc when it should have been @return
[KAFKA-505] - Remove errorcode from TopicMetaDataResponse
[KAFKA-548] - remove partition from ProducerRequestPartitionData and FetchResponsePartitionData
[KAFKA-581] - provides windows batch script for starting Kafka/Zookeeper
[KAFKA-632] - ProducerRequest should take ByteBufferMessageSet instead of MessageSet
[KAFKA-638] - remove ProducerShell
[KAFKA-667] - Rename .highwatermark file
[KAFKA-675] - Only bind to the interface declared in the 'hostname' config property
[KAFKA-699] - Disallow clients to set replicaId in FetchRequest
[KAFKA-762] - Improve second replica assignment
[KAFKA-763] - Add an option to replica from the largest offset during unclean leader election
[KAFKA-812] - Support deep iteration in DumpLogSegments tool
[KAFKA-850] - add an option to show under replicated partitions in list topic command
[KAFKA-923] - Improve controller failover latency
[KAFKA-931] - make zookeeper.connect a required property
[KAFKA-1046] - Added support for Scala 2.10 builds while maintaining compatibility with 2.8.x
Task
[KAFKA-93] - Change code header to follow standard ASF source header
[KAFKA-317] - Add support for new wire protocol to Go client
[KAFKA-341] - Create a new single host system test to validate all replicas on 0.8 branch
[KAFKA-348] - rebase 0.8 branch from trunk
[KAFKA-380] - Enhance single_host_multi_brokers test with failure to trigger leader re-election in replication
[KAFKA-440] - Create a regression test framework for distributed environment testing
[KAFKA-526] - System Test should remove the top level data log directory
[KAFKA-594] - Update System Test due to new argument "--sync" in ProducerPerformance
[KAFKA-605] - System Test - Log Retention Cases should wait longer before getting the common starting offset in replica log segments
[KAFKA-688] - System Test - Update all testcase_xxxx_properties.json for properties keys uniform naming convention
[KAFKA-737] - System Test : Disable shallow.iterator in Mirror Maker test cases to make compression work correctly
[KAFKA-791] - Fix validation bugs in System Test
[KAFKA-792] - Update multiple attributes in testcase_xxxx_properties.json
[KAFKA-819] - System Test : Add validation of log segment index to offset
[KAFKA-883] - System Test - update migration tool testsuite after 0.7 ProducerPerformance sends seq MessageID
[KAFKA-922] - System Test - set retry.backoff.ms=300 to testcase_0119
[KAFKA-932] - System Test - set retry.backoff.ms=300 to all test cases
Sub-task
[KAFKA-176] - Fix existing perf tools
[KAFKA-237] - create/delete ZK path for a topic in an admin tool
[KAFKA-239] - Wire existing producer and consumer to use the new ZK data structure
[KAFKA-240] - implement new producer and consumer request format
[KAFKA-329] - Remove the watches/broker for new topics and partitions and change create topic admin API to send start replica state change to all brokers
[KAFKA-335] - Implement an embedded controller
[KAFKA-336] - add an admin RPC to communicate state changes between the controller and the broker
[KAFKA-337] - upgrade ZKClient to allow conditional updates in ZK
[KAFKA-338] - controller failover
[KAFKA-339] - using MultiFetch in the follower
[KAFKA-342] - revisit the broker startup procedure according to V3 design
[KAFKA-343] - revisit the become leader and become follower state change operations using V3 design
[KAFKA-344] - migration tool from 0.7 to 0.8
[KAFKA-356] - Create a generic Kafka thread class that includes basic boiler plate code of instantiating and shutting down threads cleanly
[KAFKA-362] - ZookeeperConsumerConnector needs to connect to new leader after leadership change
[KAFKA-369] - remove ZK dependency on producer
[KAFKA-458] - remove errorcode from ByteBufferMessageSet
[KAFKA-482] - Make producer to run for the entire duration of the System Test
[KAFKA-488] - Port Mirroring System Test to this python system test framework
[KAFKA-492] - Sometimes the python system test framework doesn't terminate all running processes
[KAFKA-494] - Relative paths should be used for svg URLs in dashboards html
[KAFKA-502] - Simplify setup / initialization in replication_basic_test.py
[KAFKA-503] - Support "testcase_to_run" or "testcase_to_skip"
[KAFKA-507] - Shut down ZK last to avoid hanging brokers running processes
[KAFKA-513] - Add state change log to Kafka brokers
[KAFKA-571] - Add more test cases to System Test
[KAFKA-731] - ~/ivy2/cache should be a variable in the kafka-run-class bash script
[KAFKA-780] - Reassign partitions tool produces NPE in shutdown handler
[KAFKA-814] - Controller should not throw exception when a preferred replica is already the leader for a partition
[KAFKA-843] - Re-add the release-zip sbt target
Bug
[KAFKA-15] - SBT release-zip target doesn't include bin and config directories anymore
[KAFKA-42] - Support rebalancing the partitions with replication
[KAFKA-43] - Rebalance to preferred broke with intra-cluster replication support
[KAFKA-46] - Commit thread, ReplicaFetcherThread for intra-cluster replication
[KAFKA-49] - Add acknowledgement to the produce request.
[KAFKA-81] - wrong path in bin/kafka-run-class.sh
[KAFKA-97] - SocketServer.scala refers to Handler-specific variables
[KAFKA-171] - Kafka producer should do a single write to send message sets
[KAFKA-192] - CompressionUtilTest does not run and fails when it does
[KAFKA-229] - SimpleConsumer is not logging exceptions correctly so detailed stack trace is not coming in the logs
[KAFKA-233] - The producer's load balancing logic can send requests to dead brokers, when using the async producer option
[KAFKA-259] - Give better error message when trying to run shell scripts without having built/downloaded the jars yet
[KAFKA-295] - Bug in async producer DefaultEventHandler retry logic
[KAFKA-305] - SyncProducer does not correctly timeout
[KAFKA-306] - broker failure system test broken on replication branch
[KAFKA-351] - Refactor some new components introduced for replication
[KAFKA-352] - Throw exception to client if it makes a produce/consume request to a Kafka broker for a topic that hasn't been created
[KAFKA-367] - StringEncoder/StringDecoder use platform default character set
[KAFKA-370] - Exception "java.util.NoSuchElementException: None.get" appears inconsistently in Mirror Maker log.
[KAFKA-371] - Creating topic of empty string puts broker in a bad state
[KAFKA-376] - expose different data to fetch requests from the follower replicas and consumer clients
[KAFKA-379] - TopicCount.constructTopicCount isn't thread-safe
[KAFKA-385] - RequestPurgatory enhancements - expire/checkSatisfy issue; add jmx beans
[KAFKA-386] - Race condition in accessing ISR
[KAFKA-391] - Producer request and response classes should use maps
[KAFKA-396] - Mirroring system test fails on 0.8
[KAFKA-412] - deal with empty TopicData list in producer and fetch request
[KAFKA-413] - single_host_multi_brokers system test fails on laptop
[KAFKA-415] - Controller throws NoSuchElementException while marking a broker failed
[KAFKA-416] - Controller tests throw several zookeeper errors
[KAFKA-417] - NullPointerException in ZookeeperConsumerConnector
[KAFKA-418] - NullPointerException in ConsumerFetcherManager
[KAFKA-420] - maintain HW correctly with only 1 replica in ISR
[KAFKA-422] - LazyInitProducerTest has transient test failure
[KAFKA-424] - Remove invalid mirroring arguments from kafka-server-start.sh
[KAFKA-425] - Wrong class name in performance test scripts
[KAFKA-427] - LogRecoverTest.testHWCheckpointWithFailuresSingleLogSegment has transient failure
[KAFKA-428] - need to update leaderAndISR path in ZK conditionally in ReplicaManager
[KAFKA-431] - LogCorruptionTest.testMessageSizeTooLarge fails occasionally
[KAFKA-433] - ensurePartitionLeaderOnThisBroker should not access ZK
[KAFKA-434] - IllegalMonitorStateException in ReplicaManager.makerFollower
[KAFKA-453] - follower replica may need to backoff the fetching if leader is not ready yet
[KAFKA-456] - ProducerSendThread calls ListBuffer.size a whole bunch. That is a O(n) operation
[KAFKA-459] - KafkaController.RequestSendThread can throw exception on broker socket
[KAFKA-460] - ControllerChannelManager needs synchronization btw shutdown and add/remove broker
[KAFKA-461] - remove support for format for magic byte 0 in 0.8
[KAFKA-463] - log.truncateTo needs to handle targetOffset smaller than the lowest offset in the log
[KAFKA-464] - KafkaController NPE in SessionExpireListener
[KAFKA-466] - Controller tests throw IllegalStateException
[KAFKA-467] - Controller based leader election failed ERROR messages in LazyInitProducerTest
[KAFKA-468] - String#getBytes is platform dependent
[KAFKA-470] - transient unit test failure in RequestPurgatoryTest
[KAFKA-471] - Transient failure in ProducerTest
[KAFKA-473] - Use getMetadata Api in ZookeeperConsumerConnector
[KAFKA-474] - support changing host/port of a broker
[KAFKA-481] - Require values in Utils.getTopic* methods to be positive
[KAFKA-490] - Check max message size on server instead of producer
[KAFKA-491] - KafkaRequestHandler needs to handle exceptions
[KAFKA-495] - Handle topic names with "/" on Kafka server
[KAFKA-497] - recover consumer during unclean leadership change
[KAFKA-499] - Refactor controller state machine
[KAFKA-500] - javaapi support for getTopoicMetaData
[KAFKA-501] - getOfffset Api needs to return different latest offset to regular and follower consumers
[KAFKA-506] - Store logical offset in log
[KAFKA-508] - split out partiondata from fetchresponse and producerrequest
[KAFKA-509] - server should shut down on encountering invalid highwatermark file
[KAFKA-510] - broker needs to know the replication factor per partition
[KAFKA-511] - offset returned in Producer response may not be correct
[KAFKA-512] - Remove checksum from ByteBufferMessageSet.iterator
[KAFKA-514] - Replication with Leader Failure Test: Log segment files checksum mismatch
[KAFKA-516] - Consider catching all exceptions in ShutdownableThread
[KAFKA-525] - newly created partitions are not added to ReplicaStateMachine
[KAFKA-528] - IndexOutOfBoundsException thrown by kafka.consumer.ConsumerFetcherThread
[KAFKA-531] - kafka.server.ReplicaManager: java.nio.channels.NonWritableChannelException
[KAFKA-537] - expose clientId and correlationId in ConsumerConfig
[KAFKA-539] - Replica.hw should be initialized to the smaller of checkedpointed HW and log end offset
[KAFKA-540] - log.append() should halt on IOException
[KAFKA-553] - confusing reference to zk.connect in config/producer.properties
[KAFKA-556] - Change MessageSet.sizeInBytes to Int
[KAFKA-557] - Replica fetch thread doesn't need to recompute message id
[KAFKA-562] - Non-failure System Test Log Segment File Checksums mismatched
[KAFKA-563] - KafkaScheduler shutdown in ZookeeperConsumerConnector should check for config.autocommit
[KAFKA-567] - Replication Data Loss in Mirror Maker Bouncing testcase
[KAFKA-573] - System Test : Leader Failure Log Segment Checksum Mismatched When request-num-acks is 1
[KAFKA-575] - Partition.makeFollower() reads broker info from ZK
[KAFKA-576] - SimpleConsumer throws UnsupportedOperationException: empty.head
[KAFKA-577] - extend DumpLogSegments to verify consistency btw data and index
[KAFKA-578] - Leader finder thread in ConsumerFetcherManager needs to handle exceptions
[KAFKA-579] - remove connection timeout in SyncProducer
[KAFKA-580] - system test testcase_0122 under replication fails due to large # of data loss
[KAFKA-584] - produce/fetch remote time metric not set correctly when num.acks = 1
[KAFKA-586] - system test configs are broken
[KAFKA-591] - Add test cases to test log size retention and more
[KAFKA-592] - Register metrics beans at kafka server startup
[KAFKA-596] - LogSegment.firstAppendTime not reset after truncate to
[KAFKA-604] - Add missing metrics in 0.8
[KAFKA-608] - getTopicMetadata does not respect producer config settings
[KAFKA-612] - move shutting down of fetcher thread out of critical path
[KAFKA-613] - MigrationTool should disable shallow iteration in the 0.7 consumer
[KAFKA-614] - DumpLogSegment offset verification is incorrect for compressed messages
[KAFKA-618] - Deadlock between leader-finder-thread and consumer-fetcher-thread during broker failure
[KAFKA-622] - Create mbeans per client
[KAFKA-625] - Improve MessageAndMetadata to expose the partition
[KAFKA-634] - ConsoleProducer compresses messages and ignores the --compress flag
[KAFKA-646] - Provide aggregate stats at the high level Producer and ZookeeperConsumerConnector level
[KAFKA-648] - Use uniform convention for naming properties keys
[KAFKA-664] - Kafka server threads die due to OOME during long running test
[KAFKA-668] - Controlled shutdown admin tool should not require controller JMX url/port to be supplied
[KAFKA-669] - Irrecoverable error on leader while rolling to a new segment
[KAFKA-673] - Broker recovery check logic is reversed
[KAFKA-680] - ApiUtils#writeShortString uses String length instead of byte length
[KAFKA-681] - Unclean shutdown testing - truncateAndStartWithNewOffset is not invoked when it is expected to
[KAFKA-684] - ConsoleProducer does not have the queue-size option
[KAFKA-690] - TopicMetadataRequest throws exception when no topics are specified
[KAFKA-691] - Fault tolerance broken with replication factor 1
[KAFKA-692] - ConsoleConsumer outputs diagnostic message to stdout instead of stderr
[KAFKA-693] - Consumer rebalance fails if no leader available for a partition and stops all fetchers
[KAFKA-695] - Broker shuts down due to attempt to read a closed index file
[KAFKA-698] - broker may expose uncommitted data to a consumer
[KAFKA-701] - ConsoleProducer does not exit correctly and fix some config properties following KAFKA-648
[KAFKA-702] - Deadlock between request handler/processor threads
[KAFKA-708] - ISR becomes empty while marking a partition offline
[KAFKA-710] - Some arguments are always set to default in ProducerPerformance
[KAFKA-713] - Update Hadoop producer for Kafka 0.8 changes
[KAFKA-718] - kafka-run-class.sh should use reasonable gc settings
[KAFKA-726] - Add ReplicaFetcherThread name to mbean names
[KAFKA-732] - MirrorMaker with shallow.iterator.enable=true produces unreadble messages
[KAFKA-738] - correlationId is not set in FetchRequest in AbstractFetcherThread
[KAFKA-743] - PreferredReplicaLeaderElectionCommand has command line error
[KAFKA-748] - Append to index fails due to invalid offset
[KAFKA-750] - inconsistent index offset during broker startup
[KAFKA-751] - Fix windows build script - kafka-run-class.bat
[KAFKA-753] - Kafka broker shuts down while loading segments
[KAFKA-755] - standardizing json values stored in ZK
[KAFKA-756] - Processor thread blocks due to infinite loop during fetch response send
[KAFKA-757] - System Test Hard Failure cases : "Fatal error during KafkaServerStable startup" when hard-failed broker is re-started
[KAFKA-758] - startHighWaterMarksCheckPointThread is never called
[KAFKA-767] - Message Size check should be done after assigning the offsets
[KAFKA-768] - broker should exit if hitting exceptions durin startup
[KAFKA-769] - On startup, a brokers highwatermark for every topic partition gets reset to zero
[KAFKA-770] - KafkaConfig properties should be verified in the constructor
[KAFKA-772] - System Test Transient Failure on testcase_0122
[KAFKA-776] - Changing ZK format breaks some tools
[KAFKA-779] - Standardize Zk data structures for Re-assign partitions and Preferred replication election
[KAFKA-785] - Resolve bugs in PreferredReplicaLeaderElection admin tool
[KAFKA-786] - Use "withRequiredArg" while parsing jopt options in all tools
[KAFKA-793] - Include controllerId in all requests sent by controller
[KAFKA-798] - Use biased histograms instead of uniform histograms in KafkaMetricsGroup
[KAFKA-800] - inSyncReplica in Partition needs some tweaks
[KAFKA-801] - Fix MessagesInPerSec mbean to count uncompressed message rate
[KAFKA-804] - Incorrect index in the log of a follower
[KAFKA-807] - LineMessageReader doesn't correctly parse the key separator
[KAFKA-809] - Dependency on zkclient 0.1 (redundant) prevents building in IntelliJ
[KAFKA-811] - Fix clientId in migration tool
[KAFKA-813] - Minor cleanup in Controller
[KAFKA-825] - KafkaController.isActive() needs to be synchronized
[KAFKA-826] - Make Kafka 0.8 depend on metrics 2.2.0 instead of 3.x
[KAFKA-827] - improve list topic output format
[KAFKA-828] - Preferred Replica Election does not delete the admin path on controller failover
[KAFKA-830] - partition replica assignment map in the controller should be a Set
[KAFKA-832] - 0.8 Consumer prevents rebalance if consumer thread is blocked or slow
[KAFKA-840] - Controller tries to perform preferred replica election on failover before state machines have started up
[KAFKA-846] - AbstractFetcherThread should do shallow instead of deep iteration
[KAFKA-856] - Correlation id for OffsetFetch request (apache#2) always responds with 0
[KAFKA-861] - IndexOutOfBoundsException while fetching data from leader
[KAFKA-866] - Recover segment does shallow iteration to fix index causing inconsistencies
[KAFKA-871] - Rename ZkConfig properties
[KAFKA-872] - Socket server does not set send/recv buffer sizes
[KAFKA-879] - In system test, read the new leader from zookeeper instead of broker log on completion of become-leader state transition
[KAFKA-880] - NoLeaderPartitionSet should be cleared before leader finder thread is started up
[KAFKA-884] - Get java.lang.NoSuchMethodError: com.yammer.metrics.core.TimerContext.stop()J when stopping kafka brokers
[KAFKA-895] - Protocol documentation is not clear about requiredAcks = 0.
[KAFKA-899] - LeaderNotAvailableException the first time a new message for a partition is processed.
[KAFKA-900] - ClosedByInterruptException when high-level consumer shutdown normally
[KAFKA-903] - [0.8.0 - windows] FATAL - [highwatermark-checkpoint-thread1] (Logging.scala:109) - Attempt to swap the new high watermark file with the old one failed
[KAFKA-905] - Logs can have same offsets causing recovery failure
[KAFKA-907] - controller needs to close socket channel to brokers on exception
[KAFKA-914] - Deadlock between initial rebalance and watcher-triggered rebalances
[KAFKA-916] - Deadlock between fetcher shutdown and handling partitions with error
[KAFKA-919] - Disabling of auto commit is ignored during consumer group rebalancing
[KAFKA-920] - zkclient jar 0.2.0 is not compatible with 0.1.0
[KAFKA-921] - Expose max lag mbean for consumers and replica fetchers
[KAFKA-927] - Integrate controlled shutdown into kafka shutdown hook
[KAFKA-937] - ConsumerFetcherThread can deadlock
[KAFKA-938] - High CPU usage when more or less idle
[KAFKA-939] - ./sbt publish-local fails due to invalid javac flags passed to javadoc
[KAFKA-940] - Scala match error in javaapi.Implicits
[KAFKA-941] - Add Apache 2.0 license to missing code source files
[KAFKA-942] - the version of the jar should be 0.8.0-beta1 not 0.8.0-SNAPSHOT
[KAFKA-944] - the pom output from publish and publish-local is not accurate
[KAFKA-946] - Kafka Hadoop Consumer fails when verifying message checksum
[KAFKA-947] - isr-expiration-thread may block LeaderAndIsr request for a relatively long period
[KAFKA-950] - bytesSinceLastIndexEntry needs to be reset after log segment is truncated
[KAFKA-951] - Leader election rate may be reported on a non-controller
[KAFKA-953] - Remove release-zip from README we are not releasing with it
[KAFKA-954] - tidy up README file for better general availability
[KAFKA-955] - After a leader change, messages sent with ack=0 are lost
[KAFKA-956] - High-level consumer fails to check topic metadata response for errors
[KAFKA-959] - DefaultEventHandler can send more produce requests than necesary
[KAFKA-963] - when publishing to maven repository central missing signature on everything
[KAFKA-964] - Default hadoop-producer configs to request.required.acks = 1
[KAFKA-968] - Typographical Errors in Output
[KAFKA-969] - Need to prevent failure of rebalance when there are no brokers available when consumer comes up
[KAFKA-974] - can't use public release maven repo because of failure of downloaded dependency
[KAFKA-978] - kafka pom file has 2 entries for zookeeper (one with exclusion, one without)
[KAFKA-987] - Avoid checkpointing offsets in Kafka consumer that have not changed since the last commit
[KAFKA-989] - Race condition shutting down high-level consumer results in spinning background thread
[KAFKA-991] - Reduce the queue size in hadoop producer
[KAFKA-994] - High level consumer doesn't throw an exception when the message it is trying to fetch exceeds the configured fetch size
[KAFKA-1003] - ConsumerFetcherManager should pass clientId as metricsPrefix to AbstractFetcherManager
[KAFKA-1008] - Unmap before resizing
[KAFKA-1010] - Concurrency issue in getCluster() causes rebalance failure and dead consumer
[KAFKA-1017] - High number of open file handles in 0.8 producer
[KAFKA-1018] - tidy up the POM from what feedback has come from the 0.8 beta and publishing to maven
[KAFKA-1029] - Zookeeper leader election stuck in ephemeral node retry loop
[KAFKA-1030] - Addition of partitions requires bouncing all the consumers of that topic
[KAFKA-1031] - Little modification to the stop script to be able to kill the proper process
[KAFKA-1035] - Add message-send-max-retries and retry-backoff-ms options to console producer
[KAFKA-1038] - fetch response should use empty messageset instead of null when handling errors
[KAFKA-1068] - OfflinePartitionCount metrics may be incorrect after the controller failover
[KAFKA-1069] - MBean kafka.cluster.Partition report wrong UnderReplicated status
[KAFKA-1071] - The random partition selected in DefaultEventHandler is not random across producer instances
[KAFKA-1073] - CheckReassignmentStatus is broken
[KAFKA-1075] - Consumer will not rebalance upon topic partition change
[KAFKA-1076] - system tests in 0.8 are broken due to wrong log4j config
[KAFKA-1078] - Update System Test to handle controller data returned by ZK
[KAFKA-1116] - Need to upgrade sbt-assembly to compile on scala 2.10.2
[KAFKA-1126] - Remove the DISCLAIMER it is left over from incubation
[KAFKA-1131] - copy some more files into the release tar and zip that are needed/desired
[KAFKA-1133] - LICENSE and NOTICE files need to get into  META-INF when jars are built before they're signed for publishing to maven

Sub-task

[KAFKA-176] - Fix existing perf tools
[KAFKA-237] - create/delete ZK path for a topic in an admin tool
[KAFKA-239] - Wire existing producer and consumer to use the new ZK data structure
[KAFKA-240] - implement new producer and consumer request format
[KAFKA-329] - Remove the watches/broker for new topics and partitions and change create topic admin API to send start replica state change to all brokers
[KAFKA-335] - Implement an embedded controller
[KAFKA-336] - add an admin RPC to communicate state changes between the controller and the broker
[KAFKA-337] - upgrade ZKClient to allow conditional updates in ZK
[KAFKA-338] - controller failover
[KAFKA-339] - using MultiFetch in the follower
[KAFKA-342] - revisit the broker startup procedure according to V3 design
[KAFKA-343] - revisit the become leader and become follower state change operations using V3 design
[KAFKA-344] - migration tool from 0.7 to 0.8
[KAFKA-356] - Create a generic Kafka thread class that includes basic boiler plate code of instantiating and shutting down threads cleanly
[KAFKA-362] - ZookeeperConsumerConnector needs to connect to new leader after leadership change
[KAFKA-369] - remove ZK dependency on producer
[KAFKA-458] - remove errorcode from ByteBufferMessageSet
[KAFKA-482] - Make producer to run for the entire duration of the System Test
[KAFKA-488] - Port Mirroring System Test to this python system test framework
[KAFKA-492] - Sometimes the python system test framework doesn't terminate all running processes
[KAFKA-494] - Relative paths should be used for svg URLs in dashboards html
[KAFKA-502] - Simplify setup / initialization in replication_basic_test.py
[KAFKA-503] - Support "testcase_to_run" or "testcase_to_skip"
[KAFKA-507] - Shut down ZK last to avoid hanging brokers running processes
[KAFKA-513] - Add state change log to Kafka brokers
[KAFKA-571] - Add more test cases to System Test
[KAFKA-731] - ~/ivy2/cache should be a variable in the kafka-run-class bash script
[KAFKA-780] - Reassign partitions tool produces NPE in shutdown handler
[KAFKA-814] - Controller should not throw exception when a preferred replica is already the leader for a partition
[KAFKA-843] - Re-add the release-zip sbt target
Bug
[KAFKA-15] - SBT release-zip target doesn't include bin and config directories anymore
[KAFKA-42] - Support rebalancing the partitions with replication
[KAFKA-43] - Rebalance to preferred broke with intra-cluster replication support
[KAFKA-46] - Commit thread, ReplicaFetcherThread for intra-cluster replication
[KAFKA-49] - Add acknowledgement to the produce request.
[KAFKA-81] - wrong path in bin/kafka-run-class.sh
[KAFKA-97] - SocketServer.scala refers to Handler-specific variables
[KAFKA-171] - Kafka producer should do a single write to send message sets
[KAFKA-192] - CompressionUtilTest does not run and fails when it does
[KAFKA-215] - Improve system tests for the mirroring code
[KAFKA-229] - SimpleConsumer is not logging exceptions correctly so detailed stack trace is not coming in the logs
[KAFKA-259] - Give better error message when trying to run shell scripts without having built/downloaded the jars yet
[KAFKA-295] - Bug in async producer DefaultEventHandler retry logic
[KAFKA-305] - SyncProducer does not correctly timeout
[KAFKA-306] - broker failure system test broken on replication branch
[KAFKA-351] - Refactor some new components introduced for replication
[KAFKA-352] - Throw exception to client if it makes a produce/consume request to a Kafka broker for a topic that hasn't been created
[KAFKA-367] - StringEncoder/StringDecoder use platform default character set
[KAFKA-370] - Exception "java.util.NoSuchElementException: None.get" appears inconsistently in Mirror Maker log.
[KAFKA-371] - Creating topic of empty string puts broker in a bad state
[KAFKA-376] - expose different data to fetch requests from the follower replicas and consumer clients
[KAFKA-379] - TopicCount.constructTopicCount isn't thread-safe
[KAFKA-382] - Write ordering guarantee violated
[KAFKA-385] - RequestPurgatory enhancements - expire/checkSatisfy issue; add jmx beans
[KAFKA-386] - Race condition in accessing ISR
[KAFKA-391] - Producer request and response classes should use maps
[KAFKA-396] - Mirroring system test fails on 0.8
[KAFKA-412] - deal with empty TopicData list in producer and fetch request
[KAFKA-413] - single_host_multi_brokers system test fails on laptop
[KAFKA-415] - Controller throws NoSuchElementException while marking a broker failed
[KAFKA-416] - Controller tests throw several zookeeper errors
[KAFKA-418] - NullPointerException in ConsumerFetcherManager
[KAFKA-420] - maintain HW correctly with only 1 replica in ISR
[KAFKA-422] - LazyInitProducerTest has transient test failure
[KAFKA-424] - Remove invalid mirroring arguments from kafka-server-start.sh
[KAFKA-425] - Wrong class name in performance test scripts
[KAFKA-427] - LogRecoverTest.testHWCheckpointWithFailuresSingleLogSegment has transient failure
[KAFKA-428] - need to update leaderAndISR path in ZK conditionally in ReplicaManager
[KAFKA-431] - LogCorruptionTest.testMessageSizeTooLarge fails occasionally
[KAFKA-433] - ensurePartitionLeaderOnThisBroker should not access ZK
[KAFKA-434] - IllegalMonitorStateException in ReplicaManager.makerFollower
[KAFKA-452] - follower replica may need to backoff the fetching if leader is not ready yet
[KAFKA-453] - follower replica may need to backoff the fetching if leader is not ready yet
[KAFKA-456] - ProducerSendThread calls ListBuffer.size a whole bunch. That is a O(n) operation
[KAFKA-459] - KafkaController.RequestSendThread can throw exception on broker socket
[KAFKA-460] - ControllerChannelManager needs synchronization btw shutdown and add/remove broker
[KAFKA-461] - remove support for format for magic byte 0 in 0.8
[KAFKA-463] - log.truncateTo needs to handle targetOffset smaller than the lowest offset in the log
[KAFKA-464] - KafkaController NPE in SessionExpireListener
[KAFKA-466] - Controller tests throw IllegalStateException
[KAFKA-467] - Controller based leader election failed ERROR messages in LazyInitProducerTest
[KAFKA-468] - String#getBytes is platform dependent
[KAFKA-470] - transient unit test failure in RequestPurgatoryTest
[KAFKA-471] - Transient failure in ProducerTest
[KAFKA-473] - Use getMetadata Api in ZookeeperConsumerConnector
[KAFKA-474] - support changing host/port of a broker
[KAFKA-481] - Require values in Utils.getTopic* methods to be positive
[KAFKA-490] - Check max message size on server instead of producer
[KAFKA-491] - KafkaRequestHandler needs to handle exceptions
[KAFKA-495] - Handle topic names with "/" on Kafka server
[KAFKA-497] - recover consumer during unclean leadership change
[KAFKA-499] - Refactor controller state machine
[KAFKA-500] - javaapi support for getTopoicMetaData
[KAFKA-501] - getOfffset Api needs to return different latest offset to regular and follower consumers
[KAFKA-506] - Store logical offset in log
[KAFKA-508] - split out partiondata from fetchresponse and producerrequest
[KAFKA-509] - server should shut down on encountering invalid highwatermark file
[KAFKA-510] - broker needs to know the replication factor per partition
[KAFKA-511] - offset returned in Producer response may not be correct
[KAFKA-512] - Remove checksum from ByteBufferMessageSet.iterator
[KAFKA-514] - Replication with Leader Failure Test: Log segment files checksum mismatch
[KAFKA-516] - Consider catching all exceptions in ShutdownableThread
[KAFKA-525] - newly created partitions are not added to ReplicaStateMachine
[KAFKA-528] - IndexOutOfBoundsException thrown by kafka.consumer.ConsumerFetcherThread
[KAFKA-531] - kafka.server.ReplicaManager: java.nio.channels.NonWritableChannelException
[KAFKA-537] - expose clientId and correlationId in ConsumerConfig
[KAFKA-539] - Replica.hw should be initialized to the smaller of checkedpointed HW and log end offset
[KAFKA-540] - log.append() should halt on IOException
[KAFKA-553] - confusing reference to zk.connect in config/producer.properties
[KAFKA-556] - Change MessageSet.sizeInBytes to Int
[KAFKA-557] - Replica fetch thread doesn't need to recompute message id
[KAFKA-562] - Non-failure System Test Log Segment File Checksums mismatched
[KAFKA-563] - KafkaScheduler shutdown in ZookeeperConsumerConnector should check for config.autocommit
[KAFKA-567] - Replication Data Loss in Mirror Maker Bouncing testcase
[KAFKA-573] - System Test : Leader Failure Log Segment Checksum Mismatched When request-num-acks is 1
[KAFKA-575] - Partition.makeFollower() reads broker info from ZK
[KAFKA-576] - SimpleConsumer throws UnsupportedOperationException: empty.head
[KAFKA-577] - extend DumpLogSegments to verify consistency btw data and index
[KAFKA-578] - Leader finder thread in ConsumerFetcherManager needs to handle exceptions
[KAFKA-579] - remove connection timeout in SyncProducer
[KAFKA-580] - system test testcase_0122 under replication fails due to large # of data loss
[KAFKA-584] - produce/fetch remote time metric not set correctly when num.acks = 1
[KAFKA-586] - system test configs are broken
[KAFKA-591] - Add test cases to test log size retention and more
[KAFKA-592] - Register metrics beans at kafka server startup
[KAFKA-596] - LogSegment.firstAppendTime not reset after truncate to
[KAFKA-604] - Add missing metrics in 0.8
[KAFKA-608] - getTopicMetadata does not respect producer config settings
[KAFKA-612] - move shutting down of fetcher thread out of critical path
[KAFKA-613] - MigrationTool should disable shallow iteration in the 0.7 consumer
[KAFKA-614] - DumpLogSegment offset verification is incorrect for compressed messages
[KAFKA-618] - Deadlock between leader-finder-thread and consumer-fetcher-thread during broker failure
[KAFKA-622] - Create mbeans per client
[KAFKA-625] - Improve MessageAndMetadata to expose the partition
[KAFKA-634] - ConsoleProducer compresses messages and ignores the --compress flag
[KAFKA-646] - Provide aggregate stats at the high level Producer and ZookeeperConsumerConnector level
[KAFKA-648] - Use uniform convention for naming properties keys
[KAFKA-664] - Kafka server threads die due to OOME during long running test
[KAFKA-668] - Controlled shutdown admin tool should not require controller JMX url/port to be supplied
[KAFKA-669] - Irrecoverable error on leader while rolling to a new segment
[KAFKA-673] - Broker recovery check logic is reversed
[KAFKA-680] - ApiUtils#writeShortString uses String length instead of byte length
[KAFKA-681] - Unclean shutdown testing - truncateAndStartWithNewOffset is not invoked when it is expected to
[KAFKA-684] - ConsoleProducer does not have the queue-size option
[KAFKA-690] - TopicMetadataRequest throws exception when no topics are specified
[KAFKA-691] - Fault tolerance broken with replication factor 1
[KAFKA-692] - ConsoleConsumer outputs diagnostic message to stdout instead of stderr
[KAFKA-693] - Consumer rebalance fails if no leader available for a partition and stops all fetchers
[KAFKA-695] - Broker shuts down due to attempt to read a closed index file
[KAFKA-698] - broker may expose uncommitted data to a consumer
[KAFKA-701] - ConsoleProducer does not exit correctly and fix some config properties following KAFKA-648
[KAFKA-702] - Deadlock between request handler/processor threads
[KAFKA-708] - ISR becomes empty while marking a partition offline
[KAFKA-710] - Some arguments are always set to default in ProducerPerformance
[KAFKA-713] - Update Hadoop producer for Kafka 0.8 changes
[KAFKA-726] - Add ReplicaFetcherThread name to mbean names
[KAFKA-732] - MirrorMaker with shallow.iterator.enable=true produces unreadble messages
[KAFKA-738] - correlationId is not set in FetchRequest in AbstractFetcherThread
[KAFKA-743] - PreferredReplicaLeaderElectionCommand has command line error
[KAFKA-748] - Append to index fails due to invalid offset
[KAFKA-750] - inconsistent index offset during broker startup
[KAFKA-751] - Fix windows build script - kafka-run-class.bat
[KAFKA-753] - Kafka broker shuts down while loading segments
[KAFKA-755] - standardizing json values stored in ZK
[KAFKA-756] - Processor thread blocks due to infinite loop during fetch response send
[KAFKA-757] - System Test Hard Failure cases : "Fatal error during KafkaServerStable startup" when hard-failed broker is re-started
[KAFKA-758] - startHighWaterMarksCheckPointThread is never called
[KAFKA-767] - Message Size check should be done after assigning the offsets
[KAFKA-768] - broker should exit if hitting exceptions durin startup
[KAFKA-769] - On startup, a brokers highwatermark for every topic partition gets reset to zero
[KAFKA-770] - KafkaConfig properties should be verified in the constructor
[KAFKA-772] - System Test Transient Failure on testcase_0122
[KAFKA-776] - Changing ZK format breaks some tools
[KAFKA-779] - Standardize Zk data structures for Re-assign partitions and Preferred replication election
[KAFKA-785] - Resolve bugs in PreferredReplicaLeaderElection admin tool
[KAFKA-786] - Use "withRequiredArg" while parsing jopt options in all tools
[KAFKA-793] - Include controllerId in all requests sent by controller
[KAFKA-798] - Use biased histograms instead of uniform histograms in KafkaMetricsGroup
[KAFKA-800] - inSyncReplica in Partition needs some tweaks
[KAFKA-801] - Fix MessagesInPerSec mbean to count uncompressed message rate
[KAFKA-804] - Incorrect index in the log of a follower
[KAFKA-807] - LineMessageReader doesn't correctly parse the key separator
[KAFKA-809] - Dependency on zkclient 0.1 (redundant) prevents building in IntelliJ
[KAFKA-811] - Fix clientId in migration tool
[KAFKA-813] - Minor cleanup in Controller
[KAFKA-825] - KafkaController.isActive() needs to be synchronized
[KAFKA-826] - Make Kafka 0.8 depend on metrics 2.2.0 instead of 3.x
[KAFKA-827] - improve list topic output format
[KAFKA-828] - Preferred Replica Election does not delete the admin path on controller failover
[KAFKA-830] - partition replica assignment map in the controller should be a Set
[KAFKA-832] - 0.8 Consumer prevents rebalance if consumer thread is blocked or slow
[KAFKA-840] - Controller tries to perform preferred replica election on failover before state machines have started up
[KAFKA-846] - AbstractFetcherThread should do shallow instead of deep iteration
[KAFKA-856] - Correlation id for OffsetFetch request (apache#2) always responds with 0
[KAFKA-861] - IndexOutOfBoundsException while fetching data from leader
[KAFKA-866] - Recover segment does shallow iteration to fix index causing inconsistencies
[KAFKA-871] - Rename ZkConfig properties
[KAFKA-872] - Socket server does not set send/recv buffer sizes
[KAFKA-880] - NoLeaderPartitionSet should be cleared before leader finder thread is started up
[KAFKA-884] - Get java.lang.NoSuchMethodError: com.yammer.metrics.core.TimerContext.stop()J when stopping kafka brokers
[KAFKA-895] - Protocol documentation is not clear about requiredAcks = 0.
[KAFKA-899] - LeaderNotAvailableException the first time a new message for a partition is processed.
[KAFKA-900] - ClosedByInterruptException when high-level consumer shutdown normally
[KAFKA-903] - [0.8.0 - windows] FATAL - [highwatermark-checkpoint-thread1] (Logging.scala:109) - Attempt to swap the new high watermark file with the old one failed
[KAFKA-905] - Logs can have same offsets causing recovery failure
[KAFKA-907] - controller needs to close socket channel to brokers on exception
[KAFKA-914] - Deadlock between initial rebalance and watcher-triggered rebalances
[KAFKA-916] - Deadlock between fetcher shutdown and handling partitions with error
[KAFKA-919] - Disabling of auto commit is ignored during consumer group rebalancing
[KAFKA-920] - zkclient jar 0.2.0 is not compatible with 0.1.0
[KAFKA-921] - Expose max lag mbean for consumers and replica fetchers
[KAFKA-927] - Integrate controlled shutdown into kafka shutdown hook
[KAFKA-937] - ConsumerFetcherThread can deadlock
[KAFKA-938] - High CPU usage when more or less idle
[KAFKA-940] - Scala match error in javaapi.Implicits
[KAFKA-941] - Add Apache 2.0 license to missing code source files
[KAFKA-942] - the version of the jar should be 0.8.0-beta1 not 0.8.0-SNAPSHOT
[KAFKA-944] - the pom output from publish and publish-local is not accurate
Improvement
[KAFKA-77] - Implement "group commit" for kafka logs
[KAFKA-100] - ProducerShell should use high-level producer instead of SyncProducer
[KAFKA-133] - Publish kafka jar to a public maven repository
[KAFKA-134] - Upgrade Kafka to sbt 0.11.3
[KAFKA-139] - cross-compile multiple Scala versions and upgrade to SBT 0.12.1
[KAFKA-155] - Support graceful Decommissioning of Broker
[KAFKA-165] - Add helper script for zkCli.sh
[KAFKA-181] - Log errors for unrecognized config options
[KAFKA-187] - Add Snappy Compression as a Codec and refactor CompressionUtil and option on startup to select what the default codec
[KAFKA-193] - use by name parameter helper for logging and trait to include lazy logging and refactor code to use the new LogHelper
[KAFKA-246] - log configuration values used
[KAFKA-253] - Refactor the async producer to have only one queue instead of one queue per broker in a Kafka cluster
[KAFKA-258] - Remove broker.id from the broker.list config in the Producer
[KAFKA-267] - Enhance ProducerPerformance to generate unique random Long value for payload
[KAFKA-271] - Modify new FetchResponse object to remove the initial offset field
[KAFKA-281] - support multiple root log directories
[KAFKA-285] - Increase maximum value of log.retention.size
[KAFKA-296] - Update Go Client to new version of Go
[KAFKA-311] - Allow KafkaLog4jAppender to take in a configurable producer.type
[KAFKA-314] - Go Client Multi-produce
[KAFKA-323] - Add the ability to use the async producer in the Log4j appender
[KAFKA-324] - enforce broker.id to be a non-negative integer
[KAFKA-325] - revisit broker config in 0.8
[KAFKA-349] - Create individual "Response" types for each kind of request and wrap them with "BoundedByteBufferSend", remove "xxResponseSend" types for all requests except "FetchRequest"
[KAFKA-365] - change copyright in NOTICE to 2012
[KAFKA-366] - add jmx beans in broker to track # bytes in consumer
[KAFKA-368] - use the pig core jar from maven instead of distributing it
[KAFKA-393] - Add constructor for message which takes both byte array offset and length
[KAFKA-408] - ProducerPerformance does not work with all producer config options
[KAFKA-437] - Unused var statement in ZookeeperConsumerConnectorTest
[KAFKA-439] - @returns was used in scala doc when it should have been @return
[KAFKA-505] - Remove errorcode from TopicMetaDataResponse
[KAFKA-548] - remove partition from ProducerRequestPartitionData and FetchResponsePartitionData
[KAFKA-581] - provides windows batch script for starting Kafka/Zookeeper
[KAFKA-632] - ProducerRequest should take ByteBufferMessageSet instead of MessageSet
[KAFKA-638] - remove ProducerShell
[KAFKA-667] - Rename .highwatermark file
[KAFKA-675] - Only bind to the interface declared in the 'hostname' config property
[KAFKA-699] - Disallow clients to set replicaId in FetchRequest
[KAFKA-733] - Fat jar option for build, or override for ivy cache location
[KAFKA-762] - Improve second replica assignment
[KAFKA-763] - Add an option to replica from the largest offset during unclean leader election
[KAFKA-812] - Support deep iteration in DumpLogSegments tool
[KAFKA-850] - add an option to show under replicated partitions in list topic command
[KAFKA-931] - make zookeeper.connect a required property
New Feature
[KAFKA-50] - kafka intra-cluster replication support
[KAFKA-188] - Support multiple data directories
[KAFKA-202] - Make the request processing in kafka asynchonous
[KAFKA-203] - Improve Kafka internal metrics
[KAFKA-235] - Add a 'log.file.age' configuration parameter to force rotation of log files after they've reached a certain age
[KAFKA-429] - Expose JMX operation to set logger level dynamically
[KAFKA-475] - Time based log segment rollout
[KAFKA-545] - Add a Performance Suite for the Log subsystem
[KAFKA-546] - Fix commit() in zk consumer for compressed messages
Task
[KAFKA-93] - Change code header to follow standard ASF source header
[KAFKA-317] - Add support for new wire protocol to Go client
[KAFKA-341] - Create a new single host system test to validate all replicas on 0.8 branch
[KAFKA-348] - rebase 0.8 branch from trunk
[KAFKA-380] - Enhance single_host_multi_brokers test with failure to trigger leader re-election in replication
[KAFKA-440] - Create a regression test framework for distributed environment testing
[KAFKA-526] - System Test should remove the top level data log directory
[KAFKA-594] - Update System Test due to new argument "--sync" in ProducerPerformance
[KAFKA-605] - System Test - Log Retention Cases should wait longer before getting the common starting offset in replica log segments
[KAFKA-688] - System Test - Update all testcase_xxxx_properties.json for properties keys uniform naming convention
[KAFKA-737] - System Test : Disable shallow.iterator in Mirror Maker test cases to make compression work correctly
[KAFKA-791] - Fix validation bugs in System Test
[KAFKA-792] - Update multiple attributes in testcase_xxxx_properties.json
[KAFKA-819] - System Test : Add validation of log segment index to offset