-
Notifications
You must be signed in to change notification settings - Fork 8.9k
/
CHANGES.txt
1698 lines (1097 loc) · 60.9 KB
/
CHANGES.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
Hadoop YARN Change Log
Release 2.2.0 - 2013-10-13
INCOMPATIBLE CHANGES
YARN-1229. Define constraints on Auxiliary Service names. Change
ShuffleHandler service name from mapreduce.shuffle to
mapreduce_shuffle (Xuan Gong via sseth)
NEW FEATURES
IMPROVEMENTS
YARN-1246. Added application finish-status to ApplicationSummary for the sake
of testing given ApplicationHistoryServer is not yet ready. (Arpit Gupta via
vinodkv)
YARN-899. Added back queue level administrator-acls so that there is no
regression w.r.t 1.x. (Xuan Gong via vinodkv)
YARN-1228. Clean up Fair Scheduler configuration loading. (Sandy Ryza)
YARN-1213. Restore config to ban submitting to undeclared pools in the
Fair Scheduler. (Sandy Ryza)
YARN-1277. Added a policy based configuration for http/https in common
HttpServer and using the same in YARN - related to per project https config
support via HADOOP-10022. (Suresh Srinivas and Omkar Vinit Joshi via vinodkv)
OPTIMIZATIONS
BUG FIXES
YARN-1128. FifoPolicy.computeShares throws NPE on empty list of Schedulables
(Karthik Kambatla via Sandy Ryza)
YARN-1214. Register ClientToken MasterKey in SecretManager after it is
saved (Jian He via bikas)
YARN-49. Improve distributed shell application to work on a secure cluster.
(Vinod Kumar Vavilapalli via hitesh)
YARN-1157. Fixed ResourceManager UI to behave correctly when apps like
distributed-shell do not set tracking urls. (Xuan Gong via vinodkv)
YARN-1221. With Fair Scheduler, reserved MB reported in RM web UI increases
indefinitely (Siqi Li via Sandy Ryza)
YARN-1247. test-container-executor has gotten out of sync with the changes to
container-executor. (rvs via tucu)
YARN-1070. Fixed race conditions in NodeManager during container-kill.
(Zhijie Shen via vinodkv)
YARN-1215. Yarn URL should include userinfo. (Chuan Liu via cnauroth)
YARN-1262. TestApplicationCleanup relies on all containers assigned in a
single heartbeat (Karthik Kambatla via Sandy Ryza)
YARN-1260. Added webapp.http.address to yarn-default.xml so that default
install with https enabled doesn't have broken link on NM UI. (Omkar Vinit
Joshi via vinodkv)
YARN-1141. Updating resource requests should be decoupled with updating
blacklist (Zhijie Shen via bikas)
YARN-876. Node resource is added twice when node comes back from unhealthy
to healthy. (Peng Zhang via Sandy Ryza)
YARN-890. Ensure CapacityScheduler doesn't round-up metric for available
resources. (Xuan Gong & Hitesh Shah via acmurthy)
YARN-621. Changed YARN web app to not add paths that can cause duplicate
additions of authenticated filters there by causing kerberos replay errors.
(Omkar Vinit Joshi via vinodkv)
YARN-1236. FairScheduler setting queue name in RMApp is not working.
(Sandy Ryza)
YARN-1256. NM silently ignores non-existent service in
StartContainerRequest (Xuan Gong via bikas)
YARN-1149. NM throws InvalidStateTransitonException: Invalid event:
APPLICATION_LOG_HANDLING_FINISHED at RUNNING (Xuan Gong via hitesh)
YARN-1271. "Text file busy" errors launching containers again
(Sandy Ryza)
YARN-1131. $yarn logs command should return an appropriate error message if
YARN application is still running. (Siddharth Seth via hitesh)
YARN-1219. FSDownload changes file suffix making FileUtil.unTar() throw
exception. (Shanyu Zhao via cnauroth)
YARN-1251. TestDistributedShell#TestDSShell failed with timeout. (Xuan Gong
via hitesh)
YARN-1167. Fixed Distributed Shell to not incorrectly show empty hostname
on RM UI. (Xuan Gong via vinodkv)
YARN-1254. Fixed NodeManager to not pollute container's credentials. (Omkar
Vinit Joshi via vinodkv)
YARN-1273. Fixed Distributed-shell to account for containers that failed
to start. (Hitesh Shah via vinodkv)
YARN-1032. Fixed NPE in RackResolver. (Lohit Vijayarenu via acmurthy)
YARN-1090. Fixed CS UI to better reflect applications as non-schedulable
and not as pending. (Jian He via acmurthy)
YARN-1274. Fixed NodeManager's LinuxContainerExecutor to create user, app-dir
and log-dirs correctly even when there are no resources to localize for the
container. (Siddharth Seth via vinodkv)
YARN-1278. Fixed NodeManager to not delete local resources for apps on resync
command from RM - a bug caused by YARN-1149. (Hitesh Shah via vinodkv)
Release 2.1.1-beta - 2013-09-23
INCOMPATIBLE CHANGES
YARN-707. Added user information also in the YARN ClientToken so that AMs
can implement authorization based on incoming users. (Jason Lowe via vinodkv)
NEW FEATURES
IMPROVEMENTS
YARN-589. Expose a REST API for monitoring the fair scheduler (Sandy Ryza).
YARN-1074. Cleaned up YARN CLI application list to only display running
applications by default. (Xuan Gong via vinodkv)
YARN-1093. Corrections to Fair Scheduler documentation (Wing Yew Poon via
Sandy Ryza)
YARN-942. In Fair Scheduler documentation, inconsistency on which
properties have prefix (Akira Ajisaka via Sandy Ryza)
YARN-905. Add state filters to nodes CLI (Wei Yan via Sandy Ryza)
YARN-1083. Changed ResourceManager to fail when the expiry interval is less
than the configured node-heartbeat interval. (Zhijie Shen via vinodkv)
YARN-1081. Made a trivial change to YARN node CLI header to avoid potential
confusion. (Akira AJISAKA via vinodkv)
YARN-1034. Remove "experimental" in the Fair Scheduler documentation.
(Karthik Kambatla via Sandy Ryza)
YARN-1080. Improved help message for "yarn logs" command. (Xuan Gong via
vinodkv)
YARN-771. AMRMClient support for resource blacklisting (Junping Du via
bikas)
YARN-1117. Improved help messages for "yarn application" and "yarn node"
commands. (Xuan Gong via vinodkv)
YARN-758. Augment MockNM to use multiple cores (Karthik Kambatla via
Sandy Ryza)
YARN-1120. Made ApplicationConstants.Environment.USER definition OS neutral
as the corresponding value is now set correctly end-to-end. (Chuan Liu via
vinodkv)
YARN-1124. Modified YARN CLI application list to display new and submitted
applications together with running apps by default, following up YARN-1074.
(Xuan Gong via vinodkv)
YARN-1065. NM should provide AuxillaryService data to the container (Xuan
Gong via bikas)
YARN-696. Changed RMWebservice apps call to take in multiple application
states. (Trevor Lorimer via vinodkv)
YARN-910. Augmented auxiliary services to listen for container starts and
completions in addition to application events. (Alejandro Abdelnur via
vinodkv)
YARN-1137. Add support whitelist for system users to Yarn
container-executor.c. (rvs via tucu)
YARN-1001. Added a web-service to get statistics about per application-type
per state for consumption by downstream projects. (Zhijie Shen via vinodkv)
YARN-1203. Changed YARN web-app proxy to handle http and https URLs from
AM registration and finish correctly. (Omkar Vinit Joshi via vinodkv)
YARN-1204. Added separate configuration properties for https for RM and NM
without which servers enabled with https will also start on http ports.
(Omkar Vinit Joshi via vinodkv)
OPTIMIZATIONS
BUG FIXES
YARN-948. Changed ResourceManager to validate the release container list
before actually releasing them. (Omkar Vinit Joshi via vinodkv)
YARN-966. Fixed ContainerLaunch to not fail quietly when there are no
localized resources due to some other failure. (Zhijie Shen via vinodkv)
YARN-502. Fixed a state machine issue with RMNode inside ResourceManager
which was crashing scheduler. (Mayank Bansal via vinodkv)
YARN-573. Shared data structures in Public Localizer and Private Localizer
are not Thread safe. (Omkar Vinit Joshi via jlowe)
YARN-903. Changed ContainerManager to suppress unnecessary warnings when
stopping already stopped containers. (Omkar Vinit Joshi via vinodkv)
YARN-906. Fixed a bug in NodeManager where cancelling ContainerLaunch at
KILLING state causes that the container to hang. (Zhijie Shen via vinodkv)
YARN-994. HeartBeat thread in AMRMClientAsync does not handle runtime
exception correctly (Xuan Gong via bikas)
YARN-337. RM handles killed application tracking URL poorly (jlowe)
YARN-107. Fixed ResourceManager and clients to better handle
forceKillApplication on non-running and finished applications. (Xuan Gong
via vinodkv)
YARN-643. Fixed ResourceManager to remove all tokens consistently on app
finish. (Xuan Gong via vinodkv)
YARN-1006. Fixed broken rendering in the Nodes list web page on the RM web
UI. (Xuan Gong via vinodkv)
YARN-881. Priority#compareTo method seems to be wrong. (Jian He via bikas)
YARN-1082. Create base directories on HDFS after RM login to ensure RM
recovery doesn't fail in secure mode. (vinodkv via acmurthy)
YARN-1085. Modified YARN and MR2 web-apps to do HTTP authentication in
secure setup with kerberos. (Omkar Vinit Joshi via vinodkv)
YARN-1094. Fixed a blocker with RM restart code because of which RM crashes
when try to recover an existing app. (vinodkv)
YARN-1008. MiniYARNCluster with multiple nodemanagers, all nodes have same
key for allocations. (tucu)
YARN-602. Fixed NodeManager to not let users override some mandatory
environmental variables. (Kenji Kikushima via vinodkv)
YARN-1101. Active nodes can be decremented below 0 (Robert Parker
via tgraves)
YARN-981. Fixed YARN webapp so that /logs servlet works like before. (Jian He
via vinodkv)
YARN-1077. Fixed TestContainerLaunch test failure on Windows. (Chuan Liu via
vinodkv)
YARN-957. Fixed a bug in CapacityScheduler because of which requests that
need more than a node's total capability were incorrectly allocated on that
node causing apps to hang. (Omkar Vinit Joshi via vinodkv)
YARN-1107. Fixed a bug in ResourceManager because of which RM in secure mode
fails to restart. (Omkar Vinit Joshi via vinodkv)
YARN-1049. ContainerExistStatus should define a status for preempted
containers. (tucu)
YARN-1144. Unmanaged AMs registering a tracking URI should not be
proxy-fied. (tucu)
YARN-1152. Fixed a bug in ResourceManager that was causing clients to get
invalid client token key errors when an appliation is about to finish.
(Jason Lowe via vinodkv)
YARN-292. Fixed FifoScheduler and FairScheduler to make their applications
data structures thread safe to avoid RM crashing with
ArrayIndexOutOfBoundsException. (Zhijie Shen via vinodkv)
YARN-1025. ResourceManager and NodeManager do not load native libraries on
Windows. (cnauroth)
YARN-1176. RM web services ClusterMetricsInfo total nodes doesn't include
unhealthy nodes (Jonathan Eagles via tgraves)
YARN-1078. TestNodeManagerResync, TestNodeManagerShutdown, and
TestNodeStatusUpdater fail on Windows. (Chuan Liu via cnauroth)
YARN-1194. TestContainerLogsPage fails with native builds (Roman Shaposhnik
via jlowe)
YARN-1116. Populate AMRMTokens back to AMRMTokenSecretManager after RM
restarts (Jian He via bikas)
YARN-1189. NMTokenSecretManagerInNM is not being told when applications
have finished (Omkar Vinit Joshi via jlowe)
YARN-540. Race condition causing RM to potentially relaunch already
unregistered AMs on RM restart (Jian He via bikas)
YARN-1184. ClassCastException during preemption enforcement. (cdouglas)
Release 2.1.0-beta - 2013-08-22
INCOMPATIBLE CHANGES
YARN-396. Rationalize AllocateResponse in RM Scheduler API. (Zhijie Shen
via hitesh)
YARN-439. Flatten NodeHeartbeatResponse. (Xuan Gong via sseth)
YARN-440. Flatten RegisterNodeManagerResponse. (Xuan Gong via sseth)
YARN-536. Removed the unused objects ContainerStatus and ContainerStatus from
Container which also don't belong to the container. (Xuan Gong via vinodkv)
YARN-486. Changed NM's startContainer API to accept Container record given by
RM as a direct parameter instead of as part of the ContainerLaunchContext
record. (Xuan Gong via vinodkv)
YARN-444. Moved special container exit codes from YarnConfiguration to API
where they belong. (Sandy Ryza via vinodkv)
YARN-441. Removed unused utility methods for collections from two API
records. (Xuan Gong via vinodkv)
YARN-561. Modified NodeManager to set key information into the environment
of every container that it launches. (Xuan Gong via vinodkv)
YARN-579. Stop setting the Application Token in the AppMaster env, in
favour of the copy present in the container token field.
(Vinod Kumar Vavilapalli via sseth)
YARN-629. Make YarnRemoteException not be rooted at IOException. (Xuan Gong
via vinodkv)
YARN-633. Changed RMAdminProtocol api to throw IOException and
YarnRemoteException. (Xuan Gong via vinodkv)
YARN-632. Changed ContainerManager api to throw IOException and
YarnRemoteException. (Xuan Gong via vinodkv)
YARN-631. Changed ClientRMProtocol api to throw IOException and
YarnRemoteException. (Xuan Gong via vinodkv)
YARN-630. Changed AMRMProtocol api to throw IOException and
YarnRemoteException. (Xuan Gong via vinodkv)
YARN-615. Rename ContainerLaunchContext.containerTokens to tokens.
(Vinod Kumar Vavilapalli via sseth)
YARN-571. Remove user from ContainerLaunchContext. (Omkar Vinit Joshi via
vinodkv)
YARN-716. Making ApplicationID immutable. (Siddharth Seth via vinodkv)
YARN-684. ContainerManager.startContainer should use
ContainerTokenIdentifier instead of the entire Container.
(Vinod Kumar Vavilapalli via sseth)
YARN-735. Make ApplicationAttemptId, ContaienrId and NodeId immutable.
(Jian He via sseth)
YARN-749. Rename ResourceRequest.(get,set)HostName to
ResourceRequest.(get,set)ResourceName. (acmurthy)
YARN-720. container-log4j.properties should not refer to mapreduce
property names. (Zhijie Shen via sseth)
YARN-748. Moved BuilderUtils from yarn-common to yarn-server-common for
eventual retirement. (Jian He via vinodkv)
YARN-635. Renamed YarnRemoteException to YarnException. (Siddharth Seth via
vinodkv)
YARN-755. Renamed AllocateResponse.reboot to AllocateResponse.resync. (Bikas
Saha via vinodkv)
YARN-753. Added individual factory methods for all api protocol records and
converted the records to be abstract classes. (Jian He via vinodkv)
YARN-724. Moved ProtoBase from api.records to api.records.impl.pb. (Jian He
via vinodkv)
YARN-759. Create Command enum in AllocateResponse (bikas)
YARN-777. Removed unreferenced objects from .proto files. (Jian He via
vinodkv)
YARN-642. Removed health parameter from ResourceManager /nodes web-service
and cleaned the behaviour of the status parameter. (Sandy Ryza vid vinodkv)
YARN-530. Defined Service model strictly, implemented AbstractService for
robust subclassing and migrated yarn-common services. (Steve Loughran via
vinodkv)
YARN-746. Renamed Service.register() and Service.unregister() to
registerServiceListener() & unregisterServiceListener() respectively.
(Steve Loughran via vinodkv)
YARN-792. Moved NodeHealthStatus from yarn.api.record to
yarn.server.api.record. (Jian He via vinodkv)
YARN-806. Moved ContainerExitStatus from yarn.api to yarn.api.records. (Jian
He via vinodkv)
YARN-821. Renamed setFinishApplicationStatus to setFinalApplicationStatus in
FinishApplicationMasterRequest for consistency. (Jian He via vinodkv)
YARN-787. Removed minimum resource from RegisterApplicationMasterResponse.
(tucu via acmurthy)
YARN-829. Renamed RMTokenSelector to be RMDelegationTokenSelector. (Zhijie
Shen via vinodkv)
YARN-828. Removed the unsed YarnVersionAnnotation. (Zhijie Shen via vinodkv)
YARN-823. Moved RMAdmin from yarn.client to yarn.client.cli and renamed it to
be RMAdminCLI. (Jian He via vinodkv)
YARN-387. Renamed YARN protocols for consistency.
ClientRMProtocol -> ApplicationClientProtocol
AMRMProtocol -> ApplicationMasterProtocol
ContainerManager -> ContainerManagementProtocol
(vinodkv via acmurthy)
YARN-831. Removed minimum resource from GetNewApplicationResponse as a
follow-up to YARN-787. (Jian He via acmurthy)
YARN-824. Added static factory methods to hadoop-yarn-client interfaces.
(Jian He via acmurthy)
YARN-826. Moved Clock and SystemClock into yarn.util package. (Zhijie Shen
via vinodkv)
YARN-837. Moved yarn.ClusterInfo into MapReduce project as it doesn't belong
to YARN. (Zhijie Shen via vinodkv)
YARN-822. Renamed ApplicationToken to be AMRMToken, and similarly the
corresponding TokenSelector and SecretManager. (Omkar Vinit Joshi via vinodkv)
YARN-610. ClientToken is no longer set in the environment of the Containers.
(Omkar Vinit Joshi via vinodkv)
YARN-834. Fixed annotations for yarn-client module, reorganized packages and
clearly differentiated *Async apis. (Arun C Murthy and Zhijie Shen via
vinodkv)
YARN-840. Moved ProtoUtils to yarn.api.records.pb.impl. (Jian He via
acmurthy)
YARN-841. Move Auxiliary service to yarn-api, annotate and document it.
(vinodkv)
YARN-850. Rename getClusterAvailableResources to getAvailableResources in
AMRMClients (Jian He via bikas)
YARN-694. Starting to use NMTokens to authenticate all communication with
NodeManagers. (Omkar Vinit Joshi via vinodkv)
YARN-553. Replaced YarnClient.getNewApplication with
YarnClient.createApplication which provides a directly usable
ApplicationSubmissionContext to simplify the api. (Karthik Kambatla via
acmurthy)
YARN-851. Share NMTokens using NMTokenCache (api-based) between AMRMClient
and NMClient instead of memory based approach which is used currently. (Omkar
Vinit Joshi via vinodkv)
YARN-869. Move ResourceManagerAdministrationProtocol out of main YARN api.
(vinodkv via acmurthy)
YARN-791. Changed RM APIs and web-services related to nodes to ensure that
both are consistent with each other. (Sandy Ryza via vinodkv)
YARN-727. ClientRMProtocol.getAllApplications should accept ApplicationType as
a parameter. (Xuan Gong via hitesh)
YARN-701. Use application tokens irrespective of secure or non-secure
mode. (vinodkv via acmurthy)
YARN-918. Remove ApplicationAttemptId from
RegisterApplicationMasterRequestProto. (vinodkv via acmurthy)
YARN-926. Modified ContainerManagerProtcol APIs to take in requests for
multiple containers. (Jian He via vinodkv)
NEW FEATURES
YARN-482. FS: Extend SchedulingMode to intermediate queues.
(kkambatl via tucu)
YARN-45. Add protocol for schedulers to request containers back from
ApplicationMasters. (Carlo Curino, cdouglas)
YARN-563. Add the concept of an application-type for each application.
(Mayank Bansal via vinodkv)
HADOOP-8562. Enhancements to support Hadoop on Windows Server and Windows
Azure environments. (See breakdown of tasks below for subtasks and
contributors)
YARN-422. Add a NM Client library to help application-writers. (Zhijie Shen
via vinodkv)
YARN-392. Make it possible to specify hard locality constraints in resource
requests. (sandyr via tucu)
YARN-326. Add multi-resource scheduling to the fair scheduler.
(sandyr via tucu)
YARN-398. Make it possible to specify hard locality constraints in resource
requests for CapacityScheduler. (acmurthy)
YARN-781. Exposing LOGDIR in all containers' environment which should be used
by containers for logging purposes. (Jian He via vinodkv)
IMPROVEMENTS
YARN-347. Node CLI should show CPU info besides memory in node status.
(Junping Du via llu)
YARN-365. Change NM heartbeat handling to not generate a scheduler event
on each heartbeat. (Xuan Gong via sseth)
YARN-380. Fix yarn node -status output to be better readable. (Omkar Vinit
Joshi via vinodkv)
YARN-410. Fixed RM UI so that the new lines diagnostics for a failed app on
the per-application page are translated to html line breaks. (Omkar Vinit
Joshi via vinodkv)
YARN-198. Added a link to RM pages from the NodeManager web app. (Jian He
via vinodkv)
YARN-237. Refreshing the RM page forgets how many rows I had in my
Datatables (jian he via bobby)
YARN-481. Add AM Host and RPC Port to ApplicationCLI Status Output
(Chris Riccomini via bikas)
YARN-297. Improve hashCode implementations for PB records. (Xuan Gong via
hitesh)
YARN-417. Create AMRMClient wrapper that provides asynchronous callbacks.
(Sandy Ryza via bikas)
YARN-497. Yarn unmanaged-am launcher jar does not define a main class in
its manifest (Hitesh Shah via bikas)
YARN-469. Make scheduling mode in FS pluggable. (kkambatl via tucu)
YARN-450. Define value for * in the scheduling protocol (Zhijie Shen via
bikas)
YARN-475. Remove a unused constant in the public API -
ApplicationConstants.AM_APP_ATTEMPT_ID_ENV. (Hitesh Shah via vinodkv)
YARN-309. Changed NodeManager to obtain heart-beat interval from the
ResourceManager. (Xuan Gong via vinodkv)
YARN-447. Move ApplicationComparator in CapacityScheduler to use comparator
in ApplicationId. (Nemon Lou via vinodkv)
YARN-381. Improve fair scheduler docs. (Sandy Ryza via tomwhite)
YARN-458. YARN daemon addresses must be placed in many different configs.
(sandyr via tucu)
YARN-193. Scheduler.normalizeRequest does not account for allocation
requests that exceed maximumAllocation limits (Zhijie Shen via bikas)
YARN-479. NM retry behavior for connection to RM should be similar for
lost heartbeats (Jian He via bikas)
YARN-495. Changed NM reboot behaviour to be a simple resync - kill all
containers and re-register with RM. (Jian He via vinodkv)
YARN-514. Delayed store operations should not result in RM unavailability
for app submission (Zhijie Shen via bikas)
YARN-586. Fixed a typo in ApplicationSubmissionContext#setApplicationId.
(Zhijie Shen via vinodkv)
YARN-542. Changed the default global AM max-attempts value to be not one.
(Zhijie Shen via vinodkv)
YARN-583. Moved application level local resources to be localized under the
filecache sub-directory under application directory. (Omkar Vinit Joshi via
vinodkv)
YARN-581. Added a test to verify that app delegation tokens are restored
after RM restart. (Jian He via vinodkv)
YARN-577. Add application-progress also to ApplicationReport. (Hitesh Shah
via vinodkv)
YARN-595. Refactor fair scheduler to use common Resources. (Sandy Ryza
via tomwhite)
YARN-562. Modified NM to reject any containers allocated by a previous
ResourceManager. (Jian He via vinodkv)
YARN-591. Moved RM recovery related records out of public API as they do not
belong there. (vinodkv)
YARN-599. Refactoring submitApplication in ClientRMService and RMAppManager
to separate out various validation checks depending on whether they rely on
RM configuration or not. (Zhijie Shen via vinodkv)
YARN-618. Modified RM_INVALID_IDENTIFIER to be -1 instead of zero. (Jian He
via vinodkv)
YARN-625. Move the utility method unwrapAndThrowException from
YarnRemoteExceptionPBImpl to RPCUtil. (Siddharth Seth via vinodkv)
YARN-645. Moved RMDelegationTokenSecretManager from yarn-server-common to
yarn-server-resourcemanager where it really belongs. (Jian He via vinodkv)
YARN-651. Changed PBClientImpls of ContainerManager and RMAdmin to throw
IOExceptions also. (Xuan Gong via vinodkv)
YARN-582. Changed ResourceManager to recover Application token and client
tokens for app attempt so that RM can be restarted while preserving current
applications. (Jian He via vinodkv)
YARN-568. Add support for work preserving preemption to the FairScheduler.
(Carlo Curino and Sandy Ryza via cdouglas)
YARN-598. Add virtual cores to queue metrics. (sandyr via tucu)
YARN-634. Modified YarnRemoteException to be not backed by PB and introduced
a separate SerializedException record. (Siddharth Seth via vinodkv)
YARN-663. Changed ResourceTracker API and LocalizationProtocol API to throw
YarnRemoteException and IOException. (Xuan Gong via vinodkv)
YARN-590. Added an optional mesage to be returned by ResourceMaanger when RM
asks an RM to shutdown/resync etc so that NMs can log this message locally
for better debuggability. (Mayank Bansal via vinodkv)
YARN-617. Made ContainerTokens to be used for validation at NodeManager
also in unsecure mode to prevent AMs from faking resource requirements in
unsecure mode. (Omkar Vinit Joshi via vinodkv)
YARN-708. Moved RecordFactory classes to hadoop-yarn-api, and put some
miscellaneous fixes to the interfaces. (Siddharth Seth via vinodkv)
YARN-711. Copied BuilderUtil methods in individual API records as
BuilderUtils is going to be dismantled. (Jian He via vinodkv)
YARN-714. Added NMTokens to be sent to AMs as part of heart-beat response.
(Omkar Vinit Joshi via vinodkv)
YARN-638. Modified ResourceManager to restore RMDelegationTokens after
restarting. (Jian He via vinodkv)
YARN-660. Improve AMRMClient with matching requests (bikas)
YARN-717. Put object creation factories for Token in the class itself and
remove useless derivations for specific tokens. (Jian He via vinodkv)
YARN-756. Move Preemption* records to yarn.api where they really belong.
(Jian He via vinodkv)
YARN-750. Allow for black-listing resources in YARN API and Impl in CS
(acmurthy via bikas)
YARN-877. Support resource blacklisting for FifoScheduler.
(Junping Du via llu)
YARN-686. Flatten NodeReport. (sandyr via tucu)
YARN-737. Throw some specific exceptions directly instead of wrapping them
in YarnException. (Jian He via sseth)
YARN-731. RPCUtil.unwrapAndThrowException should unwrap remote
RuntimeExceptions. (Zhijie Shen via sseth)
YARN-600. Hook up cgroups CPU settings to the number of virtual cores
allocated. (sandyr via tucu)
YARN-648. FS: Add documentation for pluggable policy. (kkambatl via tucu)
YARN-773. Moved YarnRuntimeException from package api.yarn to
api.yarn.exceptions. (Jian He via vinodkv)
YARN-692. Creating NMToken master key on RM and sharing it with NM as a part
of RM-NM heartbeat. (Omkar Vinit Joshi via vinodkv)
YARN-782. vcores-pcores ratio functions differently from vmem-pmem ratio in
misleading way. (sandyr via tucu)
YARN-803. factor out scheduler config validation from the ResourceManager
to each scheduler implementation. (tucu)
YARN-789. Enable zero capabilities resource requests in fair scheduler.
(tucu)
YARN-639. Modified Distributed Shell application to start using the new
NMClient library. (Zhijie Shen via vinodkv)
YARN-693. Modified RM to send NMTokens on allocate call so that AMs can then
use them for authentication with NMs. (Omkar Vinit Joshi via vinodkv)
YARN-752. In AMRMClient, automatically add corresponding rack requests for
requested nodes. (sandyr via tucu)
YARN-825. Fixed javadoc and annotations for yarn-common module. (vinodkv)
YARN-833. Moved Graph and VisualizeStateMachine into yarn.state package.
(Zhijie Shen via vinodkv)
YARN-805. Fix javadoc and annotations on classes in the yarn-api
package. (Jian He via sseth)
YARN-846. Move pb Impl classes from yarn-api to yarn-common. (Jian He via
vinodkv)
YARN-827. Need to make Resource arithmetic methods accessible (Jian He via
bikas)
YARN-866. Add test for class ResourceWeights. (ywskycn via tucu)
YARN-736. Add a multi-resource fair sharing metric. (sandyr via tucu)
YARN-883. Expose Fair Scheduler-specific queue metrics. (sandyr via tucu)
YARN-569. Add support for requesting and enforcing preemption requests via
a capacity monitor. (Carlo Curino, cdouglas)
YARN-521. Augment AM - RM client module to be able to request containers
only at specific locations (Sandy Ryza via bikas)
YARN-513. Create common proxy client for communicating with RM. (Xuan Gong
& Jian He via bikas)
YARN-927. Change ContainerRequest to not have more than 1 container count
and remove StoreContainerRequest (bikas)
YARN-922. Change FileSystemRMStateStore to use directories (Jian He via
bikas)
YARN-865. RM webservices can't query based on application Types. (Xuan Gong
via hitesh)
YARN-912. Move client facing exceptions to yarn-api module. (Mayank Bansal
via vinodkv)
YARN-84. Use Builder to build RPC server. (Brandon Li via szetszwo)
YARN-1046. Disable mem monitoring by default in MiniYARNCluster. (Karthik
Kambatla via Sandy Ryza)
YARN-1045. Improve toString implementation for PBImpls. (Jian He via sseth)
OPTIMIZATIONS
YARN-512. Log aggregation root directory check is more expensive than it
needs to be. (Maysam Yabandeh via jlowe)
YARN-719. Move RMIdentifier from Container to ContainerTokenIdentifier.
(Vinod Kumar Vavilapalli via sseth)
BUG FIXES
YARN-383. AMRMClientImpl should handle null rmClient in stop()
(Hitesh Shah via sseth)
YARN-385. Add missing fields - location and #containers to
ResourceRequestPBImpl's toString(). (Sandy Ryza via sseth)
YARN-377. Use the new StringUtils methods added by HADOOP-9252 and fix
TestContainersMonitor. (Chris Nauroth via szetszwo)
YARN-391. Formatting fixes for LCEResourceHandler classes.
(Steve Loughran via sseth)
YARN-390. ApplicationCLI and NodeCLI hard-coded platform-specific line
separator causes test failures on Windows. (Chris Nauroth via suresh)
YARN-406. Fix TestRackResolver to function in networks where "host1"
resolves to a valid host. (Hitesh Shah via sseth)
YARN-376. Fixes a bug which would prevent the NM knowing about completed
containers and applications. (Jason Lowe via sseth)
YARN-196. Nodemanager should be more robust in handling connection failure
to ResourceManager when a cluster is started (Xuan Gong via hitesh)
YARN-485. TestProcfsProcessTree#testProcessTree() doesn't wait long enough
for the process to die. (kkambatl via tucu)
YARN-71. Fix the NodeManager to clean up local-dirs on restart.
(Xuan Gong via sseth)
YARN-378. Fix RM to make the AM max attempts/retries to be configurable
per application by clients. (Zhijie Shen via vinodkv)
YARN-498. Unmanaged AM launcher does not set various constants in env for
an AM, also does not handle failed AMs properly. (Hitesh Shah via bikas)
YARN-496. Fair scheduler configs are refreshed inconsistently in
reinitialize. (Sandy Ryza via tomwhite)
YARN-474. Fix CapacityScheduler to trigger application-activation when
am-resource-percent configuration is refreshed. (Zhijie Shen via vinodkv)
YARN-209. Fix CapacityScheduler to trigger application-activation when
the cluster capacity changes. (Zhijie Shen via vinodkv)
YARN-24. Nodemanager fails to start if log aggregation enabled and
namenode unavailable. (sandyr via tucu)
YARN-515. Node Manager not getting the master key. (Robert Joseph Evans
via jlowe)
YARN-382. SchedulerUtils improve way normalizeRequest sets the resource
capabilities. (Zhijie Shen via bikas)
YARN-467. Modify public distributed cache to localize files such that no
local directory hits unix file count limits and thus prevent job failures.
(Omkar Vinit Joshi via vinodkv)
YARN-101. Fix NodeManager heartbeat processing to not lose track of completed
containers in case of dropped heartbeats. (Xuan Gong via vinodkv)
YARN-538. RM address DNS lookup can cause unnecessary slowness on every JHS
page load. (sandyr via tucu)
YARN-532. Change RMAdmin and Localization client protocol PB implementations
to implement closeable so that they can be stopped when needed via
RPC.stopProxy(). (Siddharth Seth via vinodkv)
YARN-99. Modify private distributed cache to localize files such that no
local directory hits unix file count limits and thus prevent job failures.
(Omkar Vinit Joshi via vinodkv)
YARN-112. Fixed a race condition during localization that fails containers.
(Omkar Vinit Joshi via vinodkv)
YARN-534. Change RM restart recovery to also account for AM max-attempts
configuration after the restart. (Jian He via vinodkv)
YARN-539. Addressed memory leak of LocalResource objects NM when a resource
localization fails. (Omkar Vinit Joshi via vinodkv)
YARN-319. Submitting a job to a fair scheduler queue for which the user
does not have permission causes the client to wait forever.
(shenhong via tomwhite)
YARN-412. Fixed FifoScheduler to check hostname of a NodeManager rather
than its host:port during scheduling which caused incorrect locality for
containers. (Roger Hoover via acmurthy)
YARN-500. Fixed YARN webapps to not roll-over ports when explicitly asked
to use non-ephemeral ports. (Kenji Kikushima via vinodkv)
YARN-518. Fair Scheduler's document link could be added to the hadoop 2.x
main doc page. (sandyr via tucu)
YARN-476. ProcfsBasedProcessTree info message confuses users.
(sandyr via tucu)
YARN-585. Fix failure in TestFairScheduler#testNotAllowSubmitApplication
caused by YARN-514. (Zhijie Shen via vinodkv)
YARN-547. Fixed race conditions in public and private resource localization
which used to cause duplicate downloads. (Omkar Vinit Joshi via vinodkv)
YARN-594. Update test and add comments in YARN-534 (Jian He via bikas)
YARN-549. YarnClient.submitApplication should wait for application to be
accepted by the RM (Zhijie Shen via bikas)
YARN-605. Fix failing unit test in TestNMWebServices when versionInfo has
parantheses like when running on a git checkout. (Hitesh Shah via vinodkv)
YARN-289. Fair scheduler allows reservations that won't fit on node.
(Sandy Ryza via tomwhite)
YARN-576. Modified ResourceManager to reject NodeManagers that don't satisy
minimum resource requirements. (Kenji Kikushima via vinodkv)
YARN-646. Fix two typos in Fair Scheduler user guide. (Dapeng Sun via atm)
YARN-507. Add interface visibility and stability annotations to FS
interfaces/classes. (kkambatl via tucu)
YARN-637. FS: maxAssign is not honored. (kkambatl via tucu)
YARN-655. Fair scheduler metrics should subtract allocated memory from
available memory. (sandyr via tucu)
YARN-628. Fix the way YarnRemoteException is being unrolled to extract out
the underlying exception. (Siddharth Seth via vinodkv)
YARN-695. Remove masterContainer and status unused fields from
ApplicationReportProto and fix bugs in ApplicationReportPBImpl. (Zhijie Shen
via vinodkv)
YARN-706. Fixed race conditions in TestFSDownload. (Zhijie Shen via vinodkv).
YARN-715. Fixed unit test failures - TestDistributedShell and
TestUnmanagedAMLauncher. (Vinod Kumar Vavilapalli via sseth)
YARN-578. Fixed NM to use SecureIOUtils for reading and aggregating logs.
(Omkar Vinit Joshi via vinodkv)
YARN-733. Fixed TestNMClient from failing occasionally. (Zhijie Shen via
vinodkv)
YARN-730. Fix NMClientAsync to remove completed containers. (Zhijie Shen
via acmurthy)
YARN-726. Fix queue & finish time fields in web-ui for ResourceManager.
(Mayank Bansal via acmurthy)
YARN-757. Changed TestRMRestart to use the default scheduler to avoid test
failures. (Bikas Saha via vinodkv)
YARN-742. Log aggregation causes a lot of redundant setPermission calls.
(jlowe via kihwal)
YARN-764. blank Used Resources on Capacity Scheduler page (Nemon Lou via
tgraves)
YARN-761. TestNMClientAsync fails sometimes (Zhijie Shen via bikas)
YARN-760. NodeManager throws AvroRuntimeException on failed start.
(Niranjan Singh via jlowe)
YARN-767. Initialize application metrics at RM bootup. (Jian He via
acmurthy)
YARN-700. TestInfoBlock fails on Windows because of line ending missmatch.
(Ivan Mitic via cnauroth)
YARN-117. Migrated rest of YARN to the new service model. (Steve Louhran via
vinodkv)
YARN-812. Set default logger for application summary logger to
hadoop.root.logger. (sseth via acmurthy)
YARN-848. Nodemanager does not register with RM using the fully qualified
hostname. (Hitesh Shah via sseth)
YARN-854. Fixing YARN bugs that are failing applications in secure
environment. (Omkar Vinit Joshi via vinodkv)
YARN-861. TestContainerManager is failing. (Vinod Kumar Vavilapalli via
hitesh)
YARN-874. Making common RPC to switch to not switch to simple when other
mechanisms are enabled and thus fix YARN/MR test failures after HADOOP-9421.
(Daryn Sharp and Vinod Kumar Vavilapalli via vinodkv)
YARN-845. RM crash with NPE on NODE_UPDATE (Mayank Bansal via bikas)
YARN-369. Handle ( or throw a proper error when receiving) status updates
from application masters that have not registered (Mayank Bansal &
Abhishek Kapoor via bikas)
YARN-541. getAllocatedContainers() is not returning all the allocated
containers (bikas)
YARN-763. AMRMClientAsync should stop heartbeating after receiving
shutdown from RM (Xuan Gong via bikas)
YARN-654. AMRMClient: Perform sanity checks for parameters of public
methods (Xuan Gong via bikas)"
YARN-919. Document setting default heap sizes in yarn-env.sh (Mayank
Bansal via hitesh)
YARN-795. Fair scheduler queue metrics should subtract allocated vCores from
available vCores. (ywskycn via tucu)
YARN-799. Fix CgroupsLCEResourcesHandler to use /tasks instead of
/cgroup.procs. (Chris Riccomini via acmurthy)
YARN-333. Schedulers cannot control the queue-name of an
application. (sandyr via tucu)
YARN-368. Fixed a typo in error message in Auxiliary services. (Albert Chu
via vinodkv)
YARN-295. Fixed a race condition in ResourceManager RMAppAttempt state
machine. (Mayank Bansal via vinodkv)
YARN-523. Modified a test-case to validate container diagnostics on
localization failures. (Jian He via vinodkv)
YARN-661. Fixed NM to cleanup users' local directories correctly when
starting up. (Omkar Vinit Joshi via vinodkv)
YARN-820. Fixed an invalid state transition in NodeManager caused by failing
resource localization. (Mayank Bansal via vinodkv)
YARN-62. Modified NodeManagers to avoid AMs from abusing container tokens for