forked from voldemort/voldemort
-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathrelease_notes.txt
1306 lines (1059 loc) · 52.1 KB
/
release_notes.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
Release 1.10.22 9/20/16
* Fixes BnP resiliance to colo failures
* Fixes indentation issue with python client
* Adds 'readonly.omit.port' server configuration
* Fixes data cleanup job
* Fixes discrepancy on data retention configuration
Release 1.10.21 8/10/16
* Fix admin request timeout issue because HDFSFailedlock takes long time.
* Provide chunk size suggetion for BnP jobs.
Release 1.10.20 7/28/2016
* Reduce the server log level on missing stores
Release 1.10.19 7/25/2016
* getAll Quota support
* meaningful log message, when fetch is disabled
* Run BnP store verification in parallel.
* Auto Detect Node Id from Cluster.xml
* Validate Node Id for Host Match from Cluster.xml
* Auto update Node Id on cluster.xml update
* Generate SSH script for running on all hosts in cluster.xml (experimental)
* Concurrent Modification Exception on InsufficientOperationlNodes Exception
* Log time taken in verifyOrAddStore
* Serialize rest-port on cluster.xml
Release 1.10.18 6/28/2016
* get/set/delete quota takes QuotaType instead of string
* Fix NPE when diffMessage is called from Build and Push
* Missing store, gives a recognizable error message.
* vadmin tool increased timeout
* Logging changes for BnP
* AdminClientPool for pooling adminClients.
* AdminClient change to detect if cluster is modified
* Parallel operation support for AddStore/SetQuota
* Fetch single store for BnP Store Creation
* Utility method to query only a single store
* update store validates the store definition change
Release 1.10.17 6/1/2016
* Add additional debug info to client registry
* Fix Admin Connection/Socket timeout from Bnp
* Add idle connection timeout feature for Voldemort clients.
* Don't use Properties(properties) constructor in Voldemort
Release 1.10.16 5/23/2016
* Lower the node connection errors log to debug level
* Cleanup the HA state in HDFS directory during online state transition
* Option to fetch single store in bootstrap
* Fix Client shell reports exception on closing
* Print more info when cluster metadata check fails
* Route the system store to current zone only
* Reliable way to set the quota for stores.
* Add retry for Hdfs isFile call
* Remove support for num.chunks in the Build and Push
* ReplaceNode CLI test enhancement
* Include quota check in metadata check
* DataCleanup job picks up latest information for stores
* Gradle task tar/zip fixes
Release 1.10.15 4/18/2016
* Set-Metadata null Version error
* Multi module gradle build for voldemort
* The AdminClient's verifyOrAddStore is vulnerable to connectivity issues.
Release 1.10.14 3/29/2016
* Report metrics for Scheduler and Async Service
* better error message thrown when pushing to a store with storage quota 0
* Resolved concurrency push conflict
* Added a configurable flag on the BnP side that can disable store creation
* Fixed build.gradle
* Fixed a NPE thrown by the StorageEngineService on startup with views
* Added server state checking when bringing it back
* Fixed memory leaking issue in Hdfs FileSystem when fetching
* AdminClient updates version after updating the data
* Added second checksum validation when receiving data
* Added more more aggregatedmetrics for HDFS data
Release 1.10.13 on 2/8/2016
* Unit Test fixes for ClientRequestExecutorPool
* ShadowJar with Protobuf dependency
* HdfsFetcher error message cleanup
Release 1.10.12 on 1/11/2016
* Fixed and cleaned up HadoopStoreWriterTest.
* Fixed HDFS directory size measurement in HdfsFetcher.
Release 1.10.11 on 1/8/2016
* Fixed RO data directory diverged on 1.10.7
* Fixed BnP assigns keys into correct partitions, resulting in data loss
* Fixed unknown.host unit test failure
* Fixed HadoopStoreWriter for multiple chunks bug
* Added Defensive code for BnP Partition reducer when it sees incorrect data
Release 1.10.10 on 1/6/2016
* AdminClient constructor cleanup and respects the timeout of ClientConfig
Release 1.10.9 on 1/5/2016
* Fixed a regression affecting Read-Only servers introduced in 1.10.7.
* Fixed a unit test which failed on Java 8.
* Made vadmin.sh more resilient to node failures.
Release 1.10.8 on 1/4/2016
* Validate compression when a store is pushed via BnP
* BnP HA pushes fail, when fetch is disabled on a node.
* Enhanced Meta check which compares the file name and sizes
* Wait for Scheduler Service to shutdown, before proceeding with shutdown
* JMX errors during register/unregister for RO Stores
Release 1.10.7 on 12/15/2015
* Introduced 'build.primary.replicas.only' mode in BnP.
* Performance tool can now load custom workloads from external files.
* Fixed a bug where the server would fail to start if Bouncy Castle is
absent from the classpath.
* Admin operations improvements:
* Better error message when server receives unknown admin operation.
* Improved the Meta Check operation's default behavior.
* BnP fetches now abort immediately when their async operation is killed.
Release 1.10.6 on 12/08/2015
* BnP loggin improvements
* Option to use BouncyCastle for encryption/decryption, instead of JCE
* JMX Metrics for tracking Server state
* Enable BnP HA only on Connection errors
Release 1.10.5 on 11/25/2015
* Extended voldemort shell to let "preflist" command show parition information.
* Bug fix on python script used to generate cluster.xml.
* Bug fix updating metadata version in cluster.xml and stores.xml.
* Read-only improvements and fixes:
* Add replace URL of feching file to Voldemort node.
Release 1.10.4 on 11/11/2015
* The AdminClient in the VoldemortClientShell is now configurable.
* Improved error messages in BnP and Read-Only server-side code.
* Fixed a server-side bug in BnP HA.
* Made BnP jobs fail explicitly when clusters are inconsistent.
Release 1.10.3 on 11/04/2015
* Bug fix and improvement to the Forklift tool.
* Improved help prompt and configurability of the VoldemortClientShell.
* Moved BnP's verifyOrAddStore() to the AdminClient.
* Implement the truncate function in the RocksDB storage engine.
* N.B.: Previous data is backwards incompatible, see a0e19c6 for details.
Release 1.10.2 on 10/30/2015
* Added a ReadOnlyFileValidator script.
* Fixed rename operation in HdfsFailedFetchLock (regression fix).
Release 1.10.1 on 10/27/2015
* Read-Only server improvements and fixes:
* Read-Only server now avoids needlessly recreating symlinks.
* Individual file checksums now outputted to stats file.
* Removed a harmful set quota admin op from BnP (regression fix).
* Added more details to an exception shown in BnP (regression fix).
Release 1.10.0 on 10/19/2015
* Many many many fixes and improvements in the 1.9.x series of release! Many thanks to all contributors!
* Minor improvements since the last patch release (1.9.22):
* Upgraded RocksDB to 3.13.1 and added more configurability options for it.
* Improvements to the benchmarking tool.
* Added a unit test for colliding keys in Read-Only data files.
* Added various safe-guards in case Read-Only index/data files become corrupted.
Release 1.9.22 on 10/08/2015
* Disk quota for Voldemort Read-Only pipeline
* Improved debuggability of BnP HA
* Added heart beat for NIO selector
* Added additional keepalive settings to avoid client connection leaks
* Trimmed fat from HadoopUtils
* Proper interruption of BnP hooks
Release 1.9.21 on 09/24/2015
* Kerberos token caching now considered experimental (disabled by default).
Release 1.9.20 on 09/14/2015
* BnP/RO improvements:
* Various debuggability improvements.
* Graceful recovery from incomplete store creation.
* HA locking now happens on the server-side.
* Removed 'node.id' BnP config which caused a SPoF.
* Script improvements:
* voldemort-shell.sh properly print Avro data.
* Forklifting makes a proper schema check.
* Binary fetch-entries format.
* Unit tests improvements:
* Longer retry times on ServerTestUtils.startVoldemortServer().
* Ignored failing RocksDB test since it is unimplemented.
* New forklifting tests.
* RocksDB iteration logic is now properly implemented.
* Improved JMX beans management.
* Fixed the routing strategy of the voldsys$_store_quotas store.
Release 1.9.19 on 08/12/2015
* Removed unused things (scala shell, ec2-testing, public-lib directory)
* Fixed some unit tests.
* Various BnP improvements:
* Improved logging.
* Added bnpJar gradle target for bundling BnP with its dependencies.
* Added run-bnp.sh script for running without Azkaban.
* Various Read-Only server-side improvements:
* Upgrade to Hadoop 2.
* Kerberos authentication clean ups.
* Got rid of unused File Descriptor and other MemLock clean ups.
* Added STORAGE_SPACE quota API (not enforced yet).
Release 1.9.18 on 07/20/2015
* Various BnP improvements:
* Introduces of a limited form of High Availability for BnP.
* Improves path handling and validation in VoldemortSwapJob.
* Fetch throttling now works with block-level compression.
* Pushes to multiple colos now happen in parallel.
* Some code refactoring and clean ups.
* Minor bug fixes in scripts:
* Fixes a SecurityException when running HadoopStoreJobRunner in Oozie.
* Fixes Coordinator start script.
* EventThrottler code now uses Tehuti.
Release 1.9.17 on 06/12/2015
* Robust handling of connection failures
* Flush the queue on marking a node down
* Async connect exceptions are remembered based on connection timeout
Release 1.9.16 on 06/09/2015
* Fixes unregistration of JMX mbean.
* Fixes VoldemortConfig bug introduced in 1.9.15.
* Fixes error reporting in AvroUtils.getSchemaFromPath().
* Caches the result of duplicate AvroUtils and HadoopUtils calls in BnP.
Release 1.9.15 on 06/05/2015
* Unit tests for Serialization
* Alert if BnP jobs does not progress
* Add Timeout for ConfigurableSocketStoreFactory
* HdfsCopyStats outputs object ids, not Hdfs Paths
* HdfsCopyStatsTest fails intermittently
Release 1.9.14 on 05/22/2015
* RO stats feature
* HdfsFetcher refactoring
* Hadoop Core and IO commons version upgrade
* RO fetcher GC performance reduction by 90%
* Fix quota redistribution on zone shrinkage issue
* Admin command for meta get ro
* Store delete command fix
* Unit test fixes
Release 1.9.13 on 05/01/2015
* Non blocking connect and Non blocking Protocol negotiation.
* Suppress logging ObsoleteVersionExceptions in request handler.
* Java minimum version is bumped to Java 6.
Release 1.9.12 on 05/01/2015
* quota reset on rebalance
* Fully disabled ant build
* fix rebalance unit test intermittent failures
* Changing buffer size of GZIP streams
* Additional test cases
Release 1.9.11 on 04/23/2015
* BuildAndPush job produces compressed files as dictated by Server
* Voldemort RO Server supports on the fly decompression when compression is
enabled
* Admin request to query Server on supported compression codec
Release 1.9.10 on 04/15/2015
* Share Read/Write buffers for Client
* Share Read/Write buffers for Server on Client Operations
* Separate Client and Admin Request Handlers
* Remove extra allocations on Server side while parsing the request
* Pre-allocate Put request buffer before writing the request
* Remove one extra byte array allocation on Put, when parsing
Vector clocks from the Value
Release 1.9.9 on 04/10/2015
* Admin tools improvements and bug fixes:
* vadmin.sh now deals gracefully with inconsistent metadata and other issues.
* ReplaceNodeCLI now deals gracefully with empty stores.xml.
* ReadOnlyReplicationHelperCLI can now act on the local node, regardless of server version.
* HdfsFetcher imprpvements:
* Can fetch individual files to specific local dir (when used from CLI).
* Lower memory consumption.
* BnP improvements:
* Added a min.number.of.records config (defaults to 1) to prevent pushing empty stores.
* Improved error handling and reporting in BnP's run function.
* Client-related automated tests expanded and improved.
* Metadata queries no longer always hit the same node.
* Initial release of the RocksDB storage engine! (N.B.: still considered experimental).
Release 1.9.8 on 03/17/2015
* Fixes a bug in the vadmin restore-from-replica command.
* Improves the vadmin meta check command.
* Improves the build's tar target.
* Adds better logging in the expansion code.
* Cleans up BnP:
* Removes Azkaban dependency as much as possible.
* Standardizes on using voldemort.utils.Props as much as possible and adds more utility functions to it.
* Deletes VoldemortMultiStoreBuildAndPushJob which is not actively used and suffering from code rot.
* Adds Content-Length header support in BnP HttpHook.
* Adds safeguards in BnP HttpHook's concurrent code.
* Removes System.exit calls from BnP.
* Cleans up dependencies:
* Removed tusk, RestHadoopFetcher and related classes and tests.
* Now fetching libthrift, catalina-ant and tehuti from Maven.
Release 1.9.7 on 03/04/2015
* Improve Voldemort BuildAndPush
* Fix bugs in FailureDetector and AdminConnectionVerifier for slop streaming
* Fix log messages and English typo
* Fix NIO client thread, selector and admin thread names
Release 1.9.6 on 02/13/2015
* Bug fix for diverging metadata versions.
* IntelliJ support in gradle build.
* Avoid extra allocation on AvroGenericVersioned.
* Unit test fixes.
* Dependency upgrades:
* Jackson moved to 1.9.13
* Azkaban moved to 2.5.0, fetched from Maven Central, and removed from private-lib.
* Build and Push clean ups and improvements:
* Cleaned up config strings into constants.
* Sanity checks and error messages for bad avro configs.
* Removed proprietary LinkedIn monitoring code.
* Added open-source friendly hooks for running arbitrary code in BnP.
Release 1.9.5 on 01/12/2015
* Version metadata key fix for incorrectly incrementing on each restarts
* ReplaceNodeCLI fixes and adding post conditions.
* Tests refactoring and a new class ClientTrafficGenerator for sending traffic
* Fixed Cluster.equals to do the node state check
* Enhanced meta check-version to do the auto analysis and give output
* Readonly replication helper tool to output, to boot strap the dead node
* Windows batch file equivalents for new command functionalities
* DeleteKeysCLI tool to delete the serialized keys from Voldemort
* Krati entries iterator bug fix
Release 1.9.4 on 12/08/2014
* Add separate failure detector for slop pusher job so that metadata slop.streaming.enabled functions.
* Rename StoreVerifier to ConnectionVerifier.
* Create AdminConnectionVerifier and AdminSlopStreamingVerifier for failure detectors under OFFLINE_SERVER.
Release 1.9.3 on 11/25/2014
* Add support for disabling the client traffic (Offline server) to do maintenance on the server.
* Add support for enabling/disabling slop streaming, partition streaming, ro fetch independently.
* VAdmin support for offline server and other switches.
* ReplaceNodeCLI command to swap a node in the cluster with other voldemort node.
* Additional logging for failure detector.
Release 1.9.2 on 10/07/2014
* Fix Coordinator related tests
Release 1.9.1 on 10/03/2014
* Admin service and client for Coordinator
* Some refactoring
* Fix getmetadata shell command
* Fix origin time in coordinator rest request
Release 1.9.0 on 09/22/2014
* Updating release notes for 1.9.0
Release 1.8.16 on 09/04/2014
* QuotaException causes delete failure.
* bdb native backup fixups
Release 1.8.15 on 09/02/2014
* Fixed double-counting stat bug.
* Merged GET/GET_ALL into READ quotas and PUT/DELETE into WRITE quotas.
* Cleaned up the slop delete code.
* Fixed the C++ build so it works on more recent OS.
Release 1.8.14 on 08/26/2014
* Tehuti 0.5 that includes a fix about Histograms
Release 1.8.13 on 08/25/2014
* Fixes duplicated error message for coordinator
* Adds zone checks
* Makes tools that depend on zone 0 to be available
* Polishes Java client example
* Makes delete op use getVersion instead of get
Release 1.8.12 on 08/11/2014
* Speeds up vector clock serialization.
* Adds a clean up task in the Gradle build that fixes an intermittent issue with builds on mac.
* Expands test configurations.
* Some fixes and clean ups in the C++ client.
* Rewrites RequestCounter so that it leverages the Tehuti metrics lib.
Release 1.8.11 on 08/01/2014
* Expose http decoder parameters as config
* Client Parallel request Exceptions are not reported to failure detector
* ClientRequest Executor has infinite timeout and log cleanup
* Threshold failure detector bug fixes
Release 1.8.10 on 07/23/2014
* Registering slops for parallel puts that fail on QuotaExceededException
Release 1.8.9 on 07/02/2014
* Adding backwards compatibility checks in AdminClient
Release 1.8.8 on 06/24/2014
* Wrap mime multipart into a mime message to ensure headers are updated in
REST ResponseSenders
* Fix transport client instantiation and HttpClientFactory shutdown
* Fix build.gradle for setting java builder property in .project file
Release 1.8.6 on 06/11/2014
* Fix related to Quota metrics (store level instead of aggregate)
Release 1.8.6 on 06/11/2014
* Fix related to quotas - checking per store instead of aggregate
Release 1.8.5 on 06/06/2014
*Gradle related changes
- empty settings file
- empty voldemort-contrib directory
- gradle wrapper to fetch required version
Release 1.8.4 on 06/03/2014
* Fix RESTClientFactory for a clean shutdown
* Add check for SocketAndStreams in sendAndReceive
* Added the ability to query a single store using AdminClient / admin tool
Release 1.8.3 on 05/27/2014
* Add debug log messages for CoordinatorService
* Fix NetworkClassLoader bug when running in Windows OS (Carlos Tasada)
* Add junit support for gradle
* Fix bind failures while running tests
* Add overwrite option to fork lift tool
* Use standard logger in the StreamingClient constructor
* [python client] Fixed reconnect in _execute_request (Søren Holbech)
* Fix the admin metadata check command and did same change for old admin command.
Release 1.8.2 on 05/09/2014
* Bug fix in MetadataStore regarding updation of routing strategy
Release 1.8.1 on 05/01/2014
* Fix AbstractStoreClientFactory for backwards compatibility
* Cleaned up logging messages
* Unit tests for big Binary data
* Refactored voldemort admin tool - Phase 1 and Phase 2
Release 1.8.0 on 04/21/2014
* Split the stores.xml into individual stores on Voldemort server
* Maintaing backwards compatibility of MetadataStore
* DefaultStoreClient bootstraps using store name (instead of stores.xml)
* New Admin tool functionality to update stores (instead of replace)
Release 1.7.3 on 04/16/2014
* Fix RESTClientFactory to return a LazyStoreClient
Release 1.7.2 on 04/15/2014
* Quota limiting related fix
* Coordinator related fix
Release 1.7.1 on 04/10/2014
* Same as 1.7.0
Release 1.7.0 on 04/09/2014
* Improves BDB coversion process and documentations
* Adds scala build support and shell (Saurabh Bhatia)
* Add removeStorageEngine call to StorageConfiguration
Release 1.6.9 on 03/19/2014
* Fix a big where StreamingClient will throw NPE if not initialized
* Fix multiple broken tests
* Modified RO Rebalance Test to cover one more case
* Adding thin client shell for Voldemort Coordinator
* Move to tusk-0.0.2
Release 1.6.8 on 02/21/2014
* Add a tool to dump BDB data to Text and the reverse
* Fix racing condition problem in CachingStoreClientFactory
* Testing, client shell and admin tool enhancements from suletm,
ahimta, bhsaurabh and Holden Caufield
* ConsistencyCheckTool enhancement
* Fix bug in Coordinator for enabling different startup ports
Release 1.6.7 on 02/07/2014
* Updating the ivy file to refer to tusk-0.0.2 (build fix for nuage)
Release 1.6.6 on 02/07/2014
* Quota management in Voldemort
* Creation of read only stores in RO2 format
* Updation of query key feature in admin tool
Release 1.6.5 on 02/04/2014
* Adding AdminClient functionality to update store definitions
Release 1.6.4 on 02/03/2014
* Fix bug when RO server NPE when rolled back
* RO bug fix caused by RW optimization code
Release 1.6.3 on 01/24/2014
* Fix KeyVersionFetcher to not depend on nodeid 0
Release 1.6.2 on 01/10/2014
* Slops stored for nodes no longer in cluster, will die automatically
* Admin command to selectively purge slops destined to nodes/zones/stores
* Admin command to atomically update cluster.xml, stores.xml in one go
* Added admin command show-routing-plan to trace routing table for a key
* ZoneClipperCLI also adjusts store definitions based on dropped zone
* Numerous tests around non contiguous node ids/zone ids
Release 1.6.1 on 12/13/2013
* Avro support for fetch keys/entries admin command
* Store Clients are now cached by default, within the same factory
* Rigoruous tests around non contiguous node ids/zone ids
* ZenStoreClient efficient resource managerment
Release 1.6.0 on 11/26/2013
NOTE: This is an open source release! This release can be downloaded here:
https://github.com/voldemort/voldemort/releases
Changes made since 1.5.9:
* Fixes to remove dependency on non contiguous node ids
* Stats for store client
Release 1.5.9 on 11/05/2013
* Bug fix in AsyncMetadataVersionManager (related to store version tracking)
Release 1.5.8 on 10/29/2013
* Replace resthdfs jar with tusk.jar
* Incorporate retries into rest hdfs fetcher
Release 1.5.7 on 10/28/2013
* Voldemort now uses BDB 5.0.88
* Support for RO fetches via RestHdfs
* Fix NPE in Streaming Client
* Retrying fetches on all exceptions w/ delay
Release 1.5.6 on 10/18/2013
* Removed getStoreClientFactoryStats from StoreClientFactory interface (fix)
Release 1.5.5 on 10/18/2013
* Client shell with avro support
* Fix bind exceptions
* Fix ObsoleteVersion Exceptions
* Build once and push to multiple clusters
* Improvements to failure detector
* Fix parallel puts
* Prevent NPE in worker thread when shutdown
* Bug fixes in repair and prune job
Release 1.5.3 on 09/19/2013
* Bug fixes (NPE during shutdown
* Client side visibility changes (Histogram, Connect exception)
* Adding PruneJob and timebased resolving mechanism to streaming client
* R2Store cleanup required for the rest client
* Monitoring improvements (aggregated BDB stats, streaming stats)
* Monitoring and Config improvements for REST service
* Removed donor based rebalancing code
* Removed replica type from the code base
* Improvements to Rebalance controller
Release 1.4.7 on 09/04/13
Changes
* Added the Zone affinity feature on the client side
* Refactoring the rest package (Rest server and Coordinator)
* Using standard Netty pipeline in the Coordinator
* Added monitoring hooks for the Rest Service
Release 1.4.6 on 07/24/2013
Changes
* Cleanup of R2Store to work with the REST server
* Modified ZenStoreClient to avoid creating SystemStores during each
* re-bootstrap
* Added slop streaming functionality
Release 1.4.4 on 07/08/2013
Changes made since 1.4.3
* Change Repartioner to swap among nodes from all specified zones
* Added rebalance shuffle, cluster expansion scripts
Release 1.4.3 on 07/03/2013
Note: Server changes are not backwards compatible. To use new
rebalancing tooling, servers must be upgraded before hand.
* Rebalance features
- Proxying / abortable rebalance
- added proxy puts which rebalance safely abortable
- improved proxy gets to check locally first and only fetch remote
if not yet local
- Repartitioning: improved algorithms to optimize partition layout
- Plan: avoids cross-zone data movement in most rebalance cases. Only
expanding into a new zone requires data movement across zones.
- Controller
- better / more accurate progress monitoring.
- 'proxy pause' before rebalancing. Ensures clients pick up new
metadata and allows performance with proxy'ing to be observed
before servers start rebalancing.
- NOTE: Donor-based rebalance is currently broken by these changes.
* Rebalance tooling
- PartitionBalanceCLI: more nuanced and verbose analysis of cluster
balance
- RepartitionerCLI: Stand alone tool to determine repartitioning
- Stand alone 'repartitioning' scripts for key use cases: new cluster,
shuffle (existing cluster), cluster expansion, and zone expansion.
- RebalancePlanCLI: Stand alone plan tool to determine cost of a
specific rebalance
- RebalanceControllerCLI: Stand-alone tool for executing a specific
rebalance (plan)
* Administrivia
- java 7 compilation
- moved to Guava from Google Collections
- better unit test coverage of zoned clusters
- added example prod configuration
- added tools directory
- .gitignore improvements
* Bug fixes
- fixed concurrency bugs in metadata store
- fixed vector clock comparisons
- fixed many NPEs
- write slops in more cases when they are needed
- KeySampler/Fetcher handle larger working sets with OOME
Changes made since 1.4.2
* Minor code fixes
* Faster creation of BaseStoreRoutingPlan objects in the fast rebalancing path
Release 1.4.2 on 06/27/2013
* Fixing costly StoreRoutingPlan object construction
Release 1.4.0 on 06/21/2013
* Zone expansion release
Release 1.3.4 on 06/19/2013
* Read Write store bug fixes
- Rewrite of InMemoryStorageEngine + config to control multiVersionPuts
- Fixed a bug in the read-repair logic which was causing unnecessary puts.
- Fixed put operation in PipelineRoutedStore to ensure slop is submitted
- Fixed VectorClock bug triggered during versioned puts
* Rebalance improvement
- Made it safe to abort rebalance
- Proxy puts are established so that 'old' partition is updated and so an aborted rebalance can "roll back" without any data loss
- Proxy gets & puts are established within the zone making improving performance during rebalance
* Coordinator
- Added monitoring capability in the Coordinator
Release 1.3.3 on 04/24/2013
* VoldemortBuildandPush
- Fixed bug with schema check
* Streaming Client
- Fixed issue with redundant callback invocation
Release 1.3.1 on 03/25/2013
* HDFSFetcher
- Fixed the bug in calculating checksums when we entere a retry loop
- refactored per file checksums
- added junit test case to simulate intermittent IO exceptions
* voldemort.client.protocol.admin.AdminClient
- Added AdminStoreClient so that AdminClient can do store operations
against specific store on a specific node.
- Added helper methods for diong put & get for specific node/store
- Added voldemort.client.protocol.admin.QueryKeyResult type to
simplify QueryKey interface
* Improved FetchStreamRequestHandler and sub-classes
- Renamed all sub-classes: 'FullScan' and 'PartitionScan' prefixes
for pertinent stream request handler implementations.
- Removed unused skipRecords parameter.
- Added recordsPerPartition parameter to fetch limited portion of
each partition.
- All logic about how many keys to fetch (return to invoker) is
server side now.
* RebalanceCLI
- Added many options to help improve the balance of (zoned) clusters.
- Analysis of the balance of a cluster is significantly more detailed.
- Fixed a bug that reduced the balance of a cluster each time it was
expanded.
- Many different algorithms for improving cluster balance are
implemented in voldemort.utils.RebalanceClusterUtils
* ConsistencyCheck & ConsistencyFixCLI
- New tools for ensuring data durability. These tools are necessary
because slop creation can fail during put operations.
- ConsistencyCheck determines which keys, if any, lack
"consistency". I.e., are present on only a subset of the expected
partitions.
- ConsistencyFix takes a list of bad (inconsistent) keys and makes
sure they are present on all expected partitions.
- ConsistencyFix also has an interface for repairing "orphaned" keys
that could result from an aborted rebalance.
* KeySamplerCLI & KeyVersionFetcherCLI
- KeySamplerCLI is a new tool that reads some number of keys for
specified partitions/stores.
- KeyVersionFetcherCLI is a new tool that, given a key and a store,
fetches the version from all nodes that host a partition that
ought to store a replica of the key's value.
- Together, KeySamplerCLI and KeyVersionFetcherCLI correctly
implement the intended functionality of the Entropy tool (for
servers that implement either FullScan and PartitionScan fetches).
- Entropy tool had been used in the past to verify a sample of keys
before and after a rebalance. Entropy tool does not work as
intended/expected. This is exacerbated by the partition aware
layouts. Instead of trying to fix the Entropy tool, these two new
tools were developed. Entropy is deprecated and will eventually be
removed from the code base.
* Substantial refactoring of helper & util methods:
- voldemort.cluster.Cluster : added helper methods
- voldemort.utils.ClusterInstance : wraps up one Cluster &
List<StoreDefinition>
- voldemort.utils.Cluster : utils for single Cluster object.
- voldemort.utils.NodeUtils : utils for Node object.
- voldemort.utils.RebalanceUtils : Many methods moved to more
appropriate helper classes
- voldemort.utils.StoreDefinitionUtils : utils for StoreDefinition
object.
- voldemort.utils.StoreInstance : wraps up one Cluster & one
StoreDefinition
* Et cetera
- ByteUtils toHexString & from HexString now rely on standard
libraries
- voldemort.client.AdminFetchTest now tests FullScan and
PartitionScan fetches
- voldemort.store.routed.ReadRepairerTest annotated all tests with
@Test
Release 1.3.0 on 03/08/2013
NOTE: This is an open source release! This release can be downloaded here:
http://github.com/voldemort/voldemort/downloads.
Changes made since 1.2.3
* VoldemortConfig and ClientConfig now contain detailed documentation
* BDB-JE defaults set to ones in prod@linkedin
* Bug fixes on kerberos support for Hadoop
Release 1.2.3 on 02/20/2013
Changes made since 1.2.2
* Added a retry loop and synchronized block while getting Hadoop FS
* Code cleanup in HdfsFetcher to make it more readable.
* Throwing explicit exceptions in HdfsFetcher instead of
returning null to be more precise in the Azkaban logs.
Release 1.2.2 on 02/19/2013
Changes made since 1.2.1
* Synchronized the streaming API
* Fixed some of the streaming API tests.
Release 1.2.1 on 0/30/2013
Changes made since 1.2.0
* Added a Streaming API and related tests.
* Refactoring of the admin client apis into functional inner classes
Release 1.2.0 on 01/21/2013
Changes made since 1.1.9
* Added an Admin API to fetch orphaned key / entries
* Improved some tests related to streaming API.
* Correcting commons-codec version in ivy file (1.4)
Release 1.1.9 on 01/15/2013
Changes made since 1.1.8
* Asynchronous socket checkout improvements
* Changed checkout behavior of KeyedResourcePool to only create new
connections when there are no resources available (rather than
creating new connections until the pool is full)
* Changed QueuedKeyedResourcePool.reset behavior to better match
KeyedResourcePool (i.e., to not cancel queued asynchronous
requests unnecessarily)
* Removed (unnecessary) synchronization primitives from keyed resource pool
* Reduce granularity of failure detector locking within ThresholdFailureDetector
* Minor features/improvements
* Less verbose logging in the face of expected exceptions and errors
* Refactored (Queued)KeyedResourcePoolTest
* Bug fixes
* Fixed possible race condition for resource creation in KeyedResourcePool
* More efficient (time & space) and simpler Histogram implementation
with improved tests
Release 1.1.8 on 01/14/2013
Changes made since release 1.1.7
* Enhanced Server Monitoring
-- Server NIO layer
-- Streaming operations to the server
-- BDB storage exception counts
* Ability to turn off BDB Checkpointing during batch modifications
* Added ability to delete old checksum files in Build and Push reducer
* Upgrade Hadoop jar to 1.0.4-p2
Release 1.1.7 on 01/03/2013
NOTE: This release is based off of release 1.1.4
Changes made since release 1.1.4
* Upgrading Hadoop jar to 1.0.2
* Added support for Kerberos authentication in HdfsFetcher
* Extra config parameters for Kerberos config and keytab file
NOTE: Release 1.1.5 and 1.1.6 are special client side releases
not based off of master. 1.1.5 was rolled back to to a weird bug.
1.1.6 is a special client side release including Auto-
bootstrapper and Versioned Avro support.
Release 1.1.4 on 11/29/2012
Changes made since release 1.1.3
* Added BDB parameters to control LRU behavior in cache & proactive cleaner migration
* Added a mlock fix for pinning the indexes of RO stores in memory
Release 1.1.3 on 11/28/2012
Changes made since release 1.1.2
* Fixed a bug in the build and push job, specifically the Mapper
that caused collisions
* Added retry mechanism with the HDFS fetcher for hftp
Release 1.1.2 on 10/31/2012
Changes made since release 1.1.1
* Reverted a change to voldemort.versioning.Versioned.getVersion() so
that a Version is returned as our clients expect.
Release 1.1.1 on 10/30/2012
Changes made since release 1.1.0
* Fixed connection leak in ClientRequestExecutorFactory
* Changed client to default to DefaultStoreClient
Release 1.1.0 on 10/19/2012
Changes made since release 1.0.0
IMPORTANT NOTE : This release has significant changes to the BDB storage layer.
Users are required to read the bin/PREUPGRADE_FOR_1_1_X_README file
thoroughly before attempting to upgrade to 1.1.0. The necessary data
conversion will be done through bin/voldemort-convert-bdb.sh
* Upgrading to JE 4.1.17
* New data format that handles conflicting updates in Voldemort more
efficiently
* Move data off heap and only use it for Index
* When scanning, evict whatever you bring in right away.
* Partition based scan api to dramatically speed up rebalancing & restore
using Partition aware scans (you exactly scan whatever you want to fetch)
* Flexible knobs to control scheduling of DataCleanupJob
Release 1.0.0 on 10/17/2012
NOTE: The large version number jump from 0.96 to 1.0.0 is to
standardize on a version number of the sort MAJOR.MINOR.PATCH. This
change is part of our effort to treat internal and open source
releases in a much more similar manner. Along these lines, release
notes for internal releases (like this one) are committed on the
master branch. We hope this improves transparency as we work towards
the next open source release.
Changes made since release 0.96
* Auto bootstrapping: ZenStoreClient and System stores
* Added server side system stores for managing metadata
* ZenStoreClient interacts with system stores
* ZenStoreClient auto bootstraps whenever cluster.xml or stores.xml changes
* Added a new routing strategy to route to all with local preference
* Added a client-registry for publishing client info and config values
* Updated LazyClientStore to try to bootstrap during Init
* Modified Failure Detector to work on a shared cluster object reference
* Avro: schema evolution and read only support
* Added new Avro serializer type that supports schema evolution
* Added Avro support to read only stores
* Added LinkedIn build-and-push Azkaban jobs to build read only stores to contrib
* Added a schema backwards compatibility check to VoldemortAdminTool and on server startup to prevent mishaps due to bad schemas
* Non-blocking IO: Fixed design flaw that blocked in the face of slow servers
* Asynchronous operations no longer do a blocking checkout to get a SocketDestination
* Added additional stats collection for better visibility into request queues
* Minor features
* Enhanced VoldemortAdminTool to update store metadata version
* Enhanced VoldemortAdminTool to work with the new system stores
* Added feature to voldemort-shell.sh to dump byte & object arrays
* Added a SlowStorageEngine for testing degraded mode performance
* Added mechanism to isolate BDB cache usage among stores
* Enhanced debug logging (for traffic analysis).
* Python client bug fixes (from pull request)
* Improved messages in request tracing
* Cleaned up help/usage messages within the client shell
* Added server config to control socket backlog
* Added "--query-keys" option to query multiple keys of multiple stores from specific node
* Added control to DataCleanupJob Frequency
* Unified jmxid as the factory across the board
* Tools
* bin/generate_cluster_xml.py to generate cluster.xml
* bin/repeat-junit.sh and bin/repeat-junit-test.sh to repeatedly run tests
* Bug fixes
* Changed getall return behavior to comply with javadoc
* Fixed a bug that caused unnecessary serial requests in getall
* HFTP performance issue bug fix (fix in byte buffer and copy process)
* Fixed a bug that prevented "--fetch-keys" and "--fetch-entries" in admin tool from showing multiple store results
* Fixed problem in sample config that prevented the server from starting
* Fixed some intermittent BindException failures across many unit tests
* Fixed some intermittent rebalance test failures
* Wrapped long running tests with timeouts
Release 0.96 on 09/05/2012
Changes made since 0.90.1
* Monitoring:
* Append cluster name to various mbeans for better stats display
* Implement average throughput in bytes
* Add BDB JE stats
* Add 95th and 99th latency tracking
* Add stats for ClientRequestExecutorPool
* Add error/exception count and max getall count
* BDB+ Data cleanup Monitoring changes
* Rebalancing:
* Donor-based rebalancing and post cleanup (see https://github.com/voldemort/voldemort/wiki/Voldemort-Donor-Based-Rebalancing for more details)
* Rebalancing integration testing framework (under test/integration/voldemort/rebalance/)
* Generate multiple cluster.xml files based on the number specified when running the tool and choose the cluster with the smallest std dev as the final-cluster.xml
* Add status output to log for updateEntries (used by rebalancing)
* Read-only pipeline:
* Add hftp and webhdfs support
* Read-only bandwidth dynamic throttler
* Add minimum throttle limit per store
* Add rollback capability to the Admin tool
* Voldemort-backed stack and index linked list impl
* Change client requests to not process responses after timeout
* Modified client request executor timeout to not factor in the NIO selector timeout
* Added BDB native backup capabalities, checksum verification and incremental backups (well tested, but not yet used in production)
* Add additional client-side tracing for debugging and consistency analytics
* Clean up logging during exception at client-side
* Security exception handling
* Add snappy to CompressionStrategyFactory
* Add configurable option to interrupt service being unscheduled
* Add logging support for tracking ScanPermit owners (for debugging purposes)
* Add a jmx terminate operation for async jobs
* Add zone option for restore from replicas
* Changing the enable.nio.connector to true by default
* Better disconnection handling for python client
* Split junit tests into a long and a short test suites
* Add separate timeouts for different operations (put, get, delete, and getAll
* Allow getAll to return partial results upon timeout
* Improved cluster generation tool
* Added log4j properties folder for junit test