forked from faisalmemon/ios-crash-dump-analysis-book
-
Notifications
You must be signed in to change notification settings - Fork 0
/
index.html
4985 lines (4802 loc) · 300 KB
/
index.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
<!DOCTYPE html>
<html xmlns="http://www.w3.org/1999/xhtml" lang="" xml:lang="">
<head>
<meta charset="utf-8" />
<meta name="generator" content="pandoc" />
<meta name="viewport" content="width=device-width, initial-scale=1.0, user-scalable=yes" />
<meta name="author" content="By Faisal Memon" />
<title>iOS Crash Dump Analysis</title>
<style>
code{white-space: pre-wrap;}
span.smallcaps{font-variant: small-caps;}
span.underline{text-decoration: underline;}
div.column{display: inline-block; vertical-align: top; width: 50%;}
div.hanging-indent{margin-left: 1.5em; text-indent: -1.5em;}
ul.task-list{list-style: none;}
.display.math{display: block; text-align: center; margin: 0.5rem auto;}
</style>
<link rel="stylesheet" href="style/gitHubStyle.css" />
<!--[if lt IE 9]>
<script src="//cdnjs.cloudflare.com/ajax/libs/html5shiv/3.7.3/html5shiv-printshiv.min.js"></script>
<![endif]-->
</head>
<body>
<header id="title-block-header">
<h1 class="title">iOS Crash Dump Analysis</h1>
<p class="author">By Faisal Memon</p>
</header>
<nav id="TOC" role="doc-toc">
<ul>
<li><a href="#disclaimer">Disclaimer</a></li>
<li><a href="#preface">Preface</a></li>
<li><a href="#acknowledgements">Acknowledgements</a></li>
<li><a href="#introduction">Introduction</a></li>
<li><a href="#quick-start">Quick Start</a>
<ul>
<li><a href="#troubleshooting">Troubleshooting</a>
<ul>
<li><a href="#missing-resource-issue">Missing resource issue</a></li>
<li><a href="#binary-compatibility-issue">Binary compatibility issue</a></li>
<li><a href="#simulator-only-issue">Simulator only issue</a></li>
<li><a href="#site-specific-issues">Site specific issues</a></li>
<li><a href="#customer-device-deployment-issues">Customer device deployment issues</a></li>
<li><a href="#locale-specific-issues">Locale specific issues</a></li>
</ul></li>
<li><a href="#the-crash-mindset">The Crash Mindset</a>
<ul>
<li><a href="#locale-based-crash">Locale based Crash</a></li>
<li><a href="#geographic-location-crash">Geographic Location Crash</a></li>
<li><a href="#bus-noise-crash">Bus Noise Crash</a></li>
</ul></li>
</ul></li>
<li><a href="#basic-concepts">Basic Concepts</a>
<ul>
<li><a href="#what-is-a-crash">What is a crash?</a></li>
<li><a href="#operating-environment-policies">Operating Environment Policies</a>
<ul>
<li><a href="#nil-handling-example">Nil Handling Example</a></li>
<li><a href="#mac-address-example">MAC Address Example</a></li>
<li><a href="#camera-example">Camera Example</a></li>
<li><a href="#lessons-learnt">Lessons Learnt</a></li>
</ul></li>
<li><a href="#application-policies">Application policies</a>
<ul>
<li><a href="#when-should-we-crash">When should we crash?</a></li>
<li><a href="#when-should-we-not-crash">When should we not crash?</a></li>
</ul></li>
<li><a href="#engineering-guidance">Engineering Guidance</a>
<ul>
<li><a href="#unit-testing-the-mac-address">Unit Testing the MAC Address</a></li>
<li><a href="#ui-testing-camera-access">UI Testing Camera access</a></li>
</ul></li>
</ul></li>
<li><a href="#tooling">Tooling</a>
<ul>
<li><a href="#overview">Overview</a></li>
<li><a href="#reverse-engineering">Reverse Engineering</a></li>
<li><a href="#class-dump-tool">Class Dump Tool</a></li>
<li><a href="#third-party-crash-reporting">Third Party Crash Reporting</a></li>
</ul></li>
<li><a href="#xcode-built-in-help">Xcode Built-In Help</a>
<ul>
<li><a href="#xcode-diagnostic-settings">Xcode Diagnostic Settings</a>
<ul>
<li><a href="#execution-methodology">Execution Methodology</a></li>
<li><a href="#analysis-methodology">Analysis Methodology</a></li>
<li><a href="#process-methodology">Process Methodology</a></li>
</ul></li>
<li><a href="#the-middle-road">The Middle Road</a></li>
</ul></li>
<li><a href="#hybrid-environments">Hybrid Environments</a>
<ul>
<li><a href="#program-structure">Program structure</a></li>
<li><a href="#paradigms">Paradigms</a></li>
<li><a href="#the-problem">The Problem</a></li>
<li><a href="#solutions">Solutions</a>
<ul>
<li><a href="#stl-solution">STL Solution</a></li>
<li><a href="#facade-solution">Facade Solution</a></li>
</ul></li>
<li><a href="#lessons-learnt-1">Lessons Learnt</a></li>
</ul></li>
<li><a href="#symbolification">Symbolification</a>
<ul>
<li><a href="#build-process">Build Process</a></li>
<li><a href="#build-settings">Build Settings</a></li>
<li><a href="#observing-a-local-crash">Observing a local crash</a></li>
<li><a href="#dsym-structure">DSYM structure</a></li>
<li><a href="#manual-symbolification">Manual Symbolification</a></li>
<li><a href="#reverse-engineering-approach">Reverse Engineering Approach</a></li>
</ul></li>
<li><a href="#the-crash-report">The Crash Report</a>
<ul>
<li><a href="#system-diagnostics">System Diagnostics</a>
<ul>
<li><a href="#extracting-system-diagnostic-information">Extracting System Diagnostic Information</a></li>
</ul></li>
<li><a href="#guided-tour-of-an-ios-crash-report">Guided tour of an iOS Crash Report</a>
<ul>
<li><a href="#ios-crash-report-header-section">iOS Crash Report Header Section</a></li>
<li><a href="#ios-crash-report-date-and-version-section">iOS Crash Report Date and Version Section</a></li>
<li><a href="#ios-crash-report-exception-section">iOS Crash Report Exception Section</a></li>
<li><a href="#ios-crash-report-filtered-syslog-section">iOS Crash Report Filtered Syslog Section</a></li>
<li><a href="#ios-crash-report-exception-backtrace-section">iOS Crash Report Exception Backtrace section</a></li>
<li><a href="#ios-crash-report-thread-section">iOS Crash Report Thread Section</a></li>
<li><a href="#ios-crash-report-thread-state-section">iOS Crash Report Thread State Section</a></li>
<li><a href="#ios-crash-report-binary-images-section">iOS Crash Report Binary Images section</a></li>
</ul></li>
<li><a href="#guided-tour-of-a-macos-crash-report">Guided tour of a macOS Crash Report</a>
<ul>
<li><a href="#macos-crash-report-header-section">macOS Crash Report Header Section</a></li>
<li><a href="#macos-crash-report-date-and-version-section">macOS Crash Report Date and Version Section</a></li>
<li><a href="#macos-duration-section">macOS Duration Section</a></li>
<li><a href="#macos-crash-report-system-integrity-section">macOS Crash Report System Integrity Section</a></li>
<li><a href="#macos-crash-report-exception-section">macOS Crash Report Exception Section</a></li>
<li><a href="#macos-crash-report-thread-section">macOS Crash Report Thread Section</a></li>
<li><a href="#macos-crash-report-thread-state-section">macOS Crash Report Thread State Section</a></li>
<li><a href="#macos-crash-report-binary-images-section">macOS Crash Report Binary Images section</a></li>
<li><a href="#macos-crash-report-modification-summary">macOS Crash Report Modification Summary</a></li>
<li><a href="#macos-crash-report-virtual-memory-section">macOS Crash Report Virtual Memory Section</a></li>
<li><a href="#macos-crash-report-system-profile-section">macOS Crash Report System Profile section</a></li>
</ul></li>
</ul></li>
<li><a href="#analytic-troubleshooting">Analytic Troubleshooting</a>
<ul>
<li><a href="#prioritizing-our-problem">Prioritizing our problem</a>
<ul>
<li><a href="#prioritizing-based-upon-impact">Prioritizing based upon impact</a></li>
<li><a href="#prioritizing-based-upon-deadlines">Prioritizing based upon deadlines</a></li>
<li><a href="#prioritizing-based-upon-trend">Prioritizing based upon trend</a></li>
</ul></li>
<li><a href="#stating-the-problem">Stating the problem</a></li>
<li><a href="#specifying-the-problem">Specifying the problem</a>
<ul>
<li><a href="#questions-to-ask">Questions to ask</a></li>
</ul></li>
<li><a href="#example-problem-specifications">Example problem specifications</a>
<ul>
<li><a href="#cameraapp-what-is-is-not-example">CameraApp What Is / Is Not Example</a></li>
<li><a href="#imac-where-is-is-not-example">iMac Where Is / Is Not Example</a></li>
<li><a href="#database-app-when-is-is-not-example">Database app When Is / Is Not Example</a></li>
<li><a href="#game-app-extent-is-is-not-example">Game app Extent Is / Is Not Example</a></li>
</ul></li>
<li><a href="#the-2018-macbook-pro-t2-problem">The 2018 MacBook Pro T2 Problem</a>
<ul>
<li><a href="#analysis-of-failures">Analysis of failures</a></li>
</ul></li>
</ul></li>
<li><a href="#a-siri-crash">A Siri Crash</a>
<ul>
<li><a href="#why-are-we-looking-at-a-siri-crash">Why are we looking at a Siri Crash?</a></li>
<li><a href="#the-crash-report-1">The Crash report</a></li>
<li><a href="#the-crash-details">The Crash details</a></li>
<li><a href="#applying-our-tool-box">Applying our Tool Box</a></li>
<li><a href="#software-engineering-insights">Software Engineering Insights</a></li>
<li><a href="#lessons-learnt-2">Lessons Learnt</a></li>
</ul></li>
<li><a href="#runtime-environment-crashes">Runtime Environment Crashes</a>
<ul>
<li><a href="#unwrapping-a-nil-optional">Unwrapping a Nil Optional</a>
<ul>
<li><a href="#ios-uikit-outlets">iOS UIKit Outlets</a></li>
<li><a href="#ownership-rules">Ownership rules</a></li>
<li><a href="#crash-report-for-unwrapped-nil-optionals">Crash Report for unwrapped nil optionals</a></li>
</ul></li>
<li><a href="#releasing-a-semaphore-that-is-in-use">Releasing a semaphore that is in use</a>
<ul>
<li><a href="#crash-example-releasing-a-semaphore">Crash example releasing a semaphore</a></li>
<li><a href="#faulty-semaphore-code">Faulty semaphore code</a></li>
<li><a href="#using-application-specific-crash-report-information">Using application specific Crash Report information</a></li>
<li><a href="#semaphore-crash-lessons-learnt">Semaphore Crash Lessons Learnt</a></li>
</ul></li>
</ul></li>
<li><a href="#bad-memory-crashes">Bad Memory Crashes</a>
<ul>
<li><a href="#general-principles">General principles</a></li>
<li><a href="#segment-violation-segv-crashes">Segment Violation (SEGV) crashes</a>
<ul>
<li><a href="#fud-crash">fud crash</a></li>
<li><a href="#leakagent-crash">LeakAgent crash</a></li>
</ul></li>
<li><a href="#bus-error-sigbus-crashes">Bus error (SIGBUS) crashes</a>
<ul>
<li><a href="#xbmc-crash">xbmc crash</a></li>
<li><a href="#jablotron-crash">Jablotron crash</a></li>
</ul></li>
</ul></li>
<li><a href="#application-abort-crashes">Application Abort Crashes</a>
<ul>
<li><a href="#general-principles-1">General principles</a></li>
<li><a href="#kindle-create-crash">Kindle Create Crash</a></li>
<li><a href="#type-confusion">Type confusion</a>
<ul>
<li><a href="#extracting-nsdata-from-a-configuration-file">Extracting NSData from a configuration file</a></li>
<li><a href="#deserialization-crash-report">Deserialization Crash Report</a></li>
</ul></li>
<li><a href="#not-a-number-errors">‘Not A Number’ Errors</a></li>
</ul></li>
<li><a href="#resource-crashes">Resource Crashes</a>
<ul>
<li><a href="#cpu-usage-crash">CPU Usage Crash</a></li>
<li><a href="#wake-up-crash">Wake Up Crash</a></li>
<li><a href="#wake-up-crash-exception">Wake Up crash exception</a></li>
<li><a href="#temperature-crash">Temperature Crash</a></li>
</ul></li>
<li><a href="#application-termination-crashes">Application Termination Crashes</a>
<ul>
<li><a href="#deadlock-crash">Deadlock Crash</a></li>
<li><a href="#insecure-drawing-crash">Insecure Drawing Crash</a></li>
</ul></li>
<li><a href="#memory-diagnostics">Memory Diagnostics</a>
<ul>
<li><a href="#basics-of-memory-allocation">Basics of memory allocation</a></li>
<li><a href="#address-sanitizer">Address Sanitizer</a></li>
<li><a href="#memory-overshoot-example">Memory overshoot example</a></li>
<li><a href="#use-after-free-example">Use after free example</a></li>
<li><a href="#memory-management-tools">Memory Management tools</a>
<ul>
<li><a href="#guard-malloc-tool">Guard Malloc Tool</a></li>
<li><a href="#malloc-scribble">Malloc Scribble</a></li>
<li><a href="#zombie-objects">Zombie Objects</a></li>
<li><a href="#malloc-stack">Malloc Stack</a></li>
<li><a href="#dynamic-linker-api-usage">Dynamic Linker API Usage</a></li>
<li><a href="#dynamic-library-loads">Dynamic Library Loads</a></li>
</ul></li>
</ul></li>
<li><a href="#bibliography">Bibliography</a></li>
</ul>
</nav>
<h1 id="disclaimer">Disclaimer</h1>
<p>Copyright Faisal Memon 2018. All Rights Reserved.</p>
<p>This publication is provided “as is” without warranty of any kind, either express or implied, including, but not limited to, the implied warranties of merchantability, fitness for a particular purpose, or non-infringement.</p>
<p>This publication could include technical inaccuracies or typographical errors. Changes are periodically added to the information herein. These changes will be incorporated in new editions of the publication.</p>
<p>Apple makes no explicit or implied endorsement of this work. Materials in this book have been determined from public information sources and binaries, or materials provided by the Apple Software Development Kits.</p>
<p>Positions held by the author, as an employee or contractor, at past or future companies and institutions makes no explicit or implied endorsement of this work by those entities.</p>
<p>Every effort has been made to identify trademark terms in this text. If there is an error or omission, please contact the author. We have thus far recognised the following trademarks:</p>
<h1 id="preface">Preface</h1>
<p>This book grew from an inspiration gained back in the late 1990s.</p>
<p>It’s hard to overstate the euphoria of the moment. The Internet was the next Industrial Revolution. Investors saw it as the best place to put their money. Hardware, software and services were all experiencing a Cambrian explosion of diversity and innovation.</p>
<p>I found myself at the center of things. I was newly recruited to Sun Microsystems. At that time, if you had a great idea for a website and wanted to start serving your customers, Sun Microsystems computers were an essential purchase. These were servers that you would either house yourself or place with a co-location provider.</p>
<p>Computer technology was already well developed by then, but existing solutions were now being put to use in new scenarios, and at Internet-scale. What was a perfectly good implementation of Unix, SunOS, from the rarefied halls of Stanford University was now running on E-Bay and had to be up and running without a glitch.</p>
<p>We had an electronic board showing the status of our critical customers. Saudi Aramco was permanently lit in red to such an extent that we wondered if that was a fault on the board itself.</p>
<p>My first day was somewhat ignominious. I wasn’t even given my own cubicle. My desk looked like a school table. My keyboard had several faulty and inoperative keys. I sat in one corner of a vast cube-farm and actually forgot which corner it was at on the first day. After lunch, I returned to my desk after an extensive walk around the other corners.</p>
<p>One thing struck me was that there was a book sitting on the desk of about a quarter of the 500 odd engineers. It proudly said on the front cover, “Panic!” <span class="citation" data-cites="panicbook">(<em>Panic! Unix System Crash Dump Analysis</em> 1995)</span>. It was a book on SunOS crash dump analysis.</p>
<p>After acquiring a proper cubicle and getting to know my colleagues, I noticed that the engineers with the “Panic!” book just seemed to have that extra edge in handling low-level issues reported by Customers. Collectively it lifted the problem solving IQ of the Answer Center where I worked.</p>
<p>At Sun, there was a deep culture of learning. We were given such extensive training and support it was often the case we’d be doing seven courses per year, each a week long.</p>
<p>All was good until one course came up. It was called Analytical Troubleshooting (ATS). This caused great controversy within the Answer Center. It was a formal methodology for solving problems. It could not tell you the answers, but it would ensure you were forced to ask the right questions. It turned out that on our hardest problems, we were missing asking the right questions.</p>
<p>This was a major step forwards again in productivity. Nevertheless, some engineers, quite out of character, were loudly critical. It turned out that these techniques were just things experienced engineers had learnt as part of their craft and they didn’t want the magic to be laid bare for anyone to pick up cheaply.</p>
<p>One day Chris Drake was in town and popped into our office. He was the x86 architecture specialist that collaborated with Kimberley Brown to produce the “Panic!” book. They arranged a workshop to educate us on SunOS crashes on x86 architecture. It was something of a novelty at the time, prior to the remarkable rise of Linux and the GNU/Linux system.</p>
<p>I remember one time, as an undergraduate student during an Operating Systems lecture, I looked across the room. I noticed it was full of Sun Microsystems equipment; I stared into the Sun logo and dreamed of one day working there. It came true. So in my workshop on x86 panics, I had another idea. One day I would write a book. It would be something quite focussed on a single technical problem. It would be something that would convey the experiences I had obtained in my career. It turns out that came true as well in the book you are reading now!</p>
<h1 id="acknowledgements">Acknowledgements</h1>
<p>I’d like to acknowledge the help and support of my colleagues for writing this book.</p>
<p>Putting together this work was only possible because it was built upon generously provided open source tools, in particular <code>pandoc</code> which made writing the text of the book a pleasure.</p>
<p>Lastly, I’d like to thank my supportive family whilst I was locked in my study and largely absent. Thank you Junghyun, and Christopher.</p>
<h1 id="introduction">Introduction</h1>
<p>This book fills a gap that has emerged between Application Developers and the platform they are developing for when a crash occurs. The mindset of the Application developer is largely understanding high-level concepts and abstractions. When a crash occurs, you can often feel rudely transported into a command line UNIX world of low level constructs, pointers and raw data.</p>
<p>We focus exclusively on the Apple ecosystem.</p>
<p>We cover iOS, macOS, tvOS, watchOS and BridgeOS platforms, ARM Assembly, and C (CoreFoundation), Objective-C, and Objective-C++ and Swift programming languages. This is because the older languages are more prone to crash bugs. Real world applications tend to end up being a hybrid between the safer Swift language and older technologies.</p>
<p>We assume you have at least an introductory knowledge of iOS programming and software engineering, and have access to a Mac with Xcode.</p>
<p>The approach we take is to combine three different perspectives on the problem to give a rounded and robust view of the situation and how to resolve it.</p>
<p>Our first perspective is to deliver a HOW-TO guide for using the excellent tooling available from Apple.</p>
<p>Our second perspective is to provide a discussion of software engineering concepts tailored to preventing and resolving crashes.</p>
<p>Our third perspective is to offer a formal problem-solving approach but applied to crash analysis.</p>
<p>Programming literature comprehensively has documented software engineering concepts, and Apple has documented their crash dump tooling via Guides and WWDC videos.</p>
<p>Formal problem solving is less discussed in software engineering circles, perhaps because it’s considered a table stakes skill for an engineer. It is however a discipline of its own and when directly studied can only enhance the “natural” abilities that seem to mark out the “technically-minded” folks from the rest of the population.</p>
<p>Our goal is not the shy away from repeating knowledge we’ve probably seen or read elsewhere but instead we take the viewpoint of explaining the whole narrative in a cohesive manner. What makes crash dump analysis hard is that significant background knowledge is often assumed in order to make room to concentrate on the particulars of a specific tool or Crash Report. That causes a barrier to entry, which this book aims to overcome.</p>
<p>To complement the book, there is a website of resources which is intended to be used alongside the printed material so example projects can be setup and run by yourself and experimented with. All references in this book are collected into the Bibliography Chapter at the end of the book. There you will find URLs to resources, for example.</p>
<p>The GitHub website supporting the book is at <span class="citation" data-cites="icdabgithub">(“IOS Crash Dump Analysis Book Github Resources” 2018)</span></p>
<h1 id="quick-start">Quick Start</h1>
<p>When an application crash appears after a recent code change, it can be straightforward to reason about the crash and look at the relevant code changes. Often, crashes just appear due to a change in operating environment. Those can be the most annoying. For example, the app runs fine in the Office but crashes at the customer Site. We don’t have time to get into why, but need a quick fix or workaround. Another common problem scenario arises when a new project is being explored. This is where we have no prior experience with the code base but immediately face crash problems after compilation and running the app.</p>
<p>In this chapter, we explore possible reasons for crashing due to a change in operating environment. Many problems can be dealt with without getting into logical analysis of the specifics of the problem at hand. In reality sometimes we just need to make progress, whilst making a note to go back and address the root cause.</p>
<h2 id="troubleshooting">Troubleshooting</h2>
<h3 id="missing-resource-issue">Missing resource issue</h3>
<p>Sometimes our app crashes on startup due to a missing resource issue.</p>
<p>We should try compiling and running other Xcode targets within the same project. Sometimes a specific target is the one that sets up the needed environment as part of the build. If so, we can make a note to address that later.</p>
<h3 id="binary-compatibility-issue">Binary compatibility issue</h3>
<p>Sometimes our app crashes on startup due to a binary compatibility issue.</p>
<p>If we’ve recently updated Xcode, or pulled code updates on top of a compiled project, we can perform an Option-Command-Shift-K clean which cleans the build area of intermediates, and then re-build as normal.</p>
<h3 id="simulator-only-issue">Simulator only issue</h3>
<p>Sometimes our app crashes only on simulator.</p>
<p>Here we should try Simulator Hardware->Reset all content and settings. We can try iPad simulator instead of iPhone simulator or vice-versa. Sample projects are often used to explain a particular technology without regard to productisation or generality.</p>
<h3 id="site-specific-issues">Site specific issues</h3>
<p>Sometimes our app only crashes when at customer site.</p>
<p>We can check Wi-Fi settings or try hot-spotting our iPad to an iPhone. Sometimes network issues such as connectivity, or latency are overlooked when developing our app in the office/home environment. We should make a note to fix networking assumptions if that is the problem.</p>
<h3 id="customer-device-deployment-issues">Customer device deployment issues</h3>
<p>Sometimes our app only crashes when deployed onto a customer device.</p>
<p>If we cable up our laptop to the customer’s device, we’re probably doing a Debug release deployment. This means push notification tokens will be the development tokens not the production tokens. It also may mean that resource access grants (to Camera for example) are no longer valid as they may have been approved via a TestFlight or App Store version of the app previously (production version).</p>
<p>We should try switching deployment configuration via Command-< selecting Run in the left panel, Info tab in the right panel, Build Configuration setting Release (not Debug). We should also manually check any resource access grants in the iPad/iPhone settings.</p>
<h3 id="locale-specific-issues">Locale specific issues</h3>
<p>Sometimes deploying with the customer’s locale causes a crash.</p>
<p>Resource files might be absent in the wrong locale. Furthermore, locale handling is rife with undocumented special cases. We should try changing the locale temporarily to a known working one. Make a note to return to the issue when back in the office.</p>
<h2 id="the-crash-mindset">The Crash Mindset</h2>
<p>One take away lesson from the above examples is that we need to think of our code in a wider context. We should think of the operating environment of our app. This comprises:</p>
<ul>
<li>the compiled code</li>
<li>binary incompatibilities between code modules (different language versions, compilers and toolchains)</li>
<li>resource files bundled or downloaded into the app</li>
<li>the build configuration (e.g. Release or Debug)</li>
<li>the network environment, availability/latency/speed</li>
<li>permissions granted to the app</li>
<li>permissions denied to the app (in a Mobile Device Management secured environment)</li>
<li>platform variants</li>
<li>orientation</li>
<li>foreground and background operating modes</li>
<li>hardware performance (old slow hardware versus faster newer devices)</li>
<li>hardware components (GPU, Memory, CPU, accessories, etc.)</li>
<li>geographic location related issues</li>
<li>locale issues</li>
<li>presence of diagnostics settings</li>
<li>presence of a debugger or profiler</li>
<li>the OS version of the target device</li>
</ul>
<p>As a first step in getting into the correct mindset to tackle app crashes, its worthwhile working through each of the above operating environment differences and trying to note down if such a difference ever resulted in a crash that we know about or suspect could happen. This teaches us that crashes are much more about <strong>environment</strong> than about <strong>source code</strong>. Another secondary insight is that the more able we are to produce a list of hypotheses given a specific environment difference, the more easily and quickly we will be able to find the root cause of crashes that seem mysterious to other people, and almost magical that we came up with a suggestion of where the problem could be.</p>
<p>Here are some curious examples of crashes from the Information Technology folklore to whet our appetite and get us thinking:</p>
<h3 id="locale-based-crash">Locale based Crash</h3>
<p>The Russian locale caused a crash during date processing.</p>
<p>This was because 1984-04-01 was being used as a sentinel date marker. However, in Russia, there is no such date/time because there is no midnight at that point in time. Daylight time started in Russia on that date with a +1 hour.</p>
<p>This was seen during development of the WecudosPro iPad app when it was tested in Russia</p>
<h3 id="geographic-location-crash">Geographic Location Crash</h3>
<p>A computer was crashing each day at a different time.</p>
<p>The actual problem was the computer was near a window next to an estuary where ships passed by. At high tide, a military ship would sail past and its RADAR would disrupt the electronics and cause a crash.</p>
<p>This folklore story was told to Sun Microsystems Answer Center engineers in the UK during Kepner-Tregoe formal problem solving training.</p>
<h3 id="bus-noise-crash">Bus Noise Crash</h3>
<p>When a computer was under both heavy network load, and disk load, the system would crash.</p>
<p>The crash was due to corruption on disk. There were zeroes every 64 bytes. It was the cache line size of the computer. The memory board was not wired up correctly causing noise at 64 byte boundaries picked up by the disk ribbon cable sitting next to it.</p>
<p>This was seen in an early prototype of a Sun Volume Systems Group computer.</p>
<h1 id="basic-concepts">Basic Concepts</h1>
<h2 id="what-is-a-crash">What is a crash?</h2>
<p>Inside our computers is an Operating Environment. This comprises one or more running Operating Systems, and Application Software. Operating Systems and Application Software are distinguished by the fact that OS software runs with higher CPU privileges (kernel mode) than Application Software (user mode).</p>
<p>The basic conceptual model of our application software sitting on an Operating System which itself sits on hardware is normally sufficient. However, modern computer systems have multiple co-operating subsystems. For example, a MacBook Pro with TouchBar will have the main operating system, macOS, but also Bridge OS providing the TouchBar interface, disk encryption and “Hey Siri!” support. The multimedia and networking chips in our computers are advanced components and can have their own real-time Operating Systems running on them. Our Mac software will be just one of many applications running on macOS.</p>
<p>An application crash is something the Operating Environment does to our application in response to what we have done (or failed to do) in the Operating Environment that violates some <em>policy</em> of the platform we are running on.</p>
<p>When the Operating System detects a problem in the Operating System, it can crash itself. This is called a kernel panic.</p>
<h2 id="operating-environment-policies">Operating Environment Policies</h2>
<p>The policies of the operating environment are there to ensure security, data safety, performance, and privacy of the environment to the user.</p>
<h3 id="nil-handling-example">Nil Handling Example</h3>
<p>Newcomers to the Apple ecosystem are often surprised to learn that Objective-C allows us to message a nil object. It silently ignores the failed dispatch. For example, the following method runs ok.</p>
<pre><code>- (void)nilDispatchDoesNothing
{
NSString *error = NULL;
assert([error length] == 0);
}</code></pre>
<p>The Objective-C runtime authors made a judgement call, and decided it was better for an application to ignore such problems.</p>
<p>However if we dereference a C pointer we get a crash.</p>
<pre><code>void nullDereferenceCrash() {
char *nullPointer = NULL;
assert(strlen(nullPointer) == 0);
}</code></pre>
<p>The authors of the operating system have setup the system so access to this and other low memory addresses causes the hardware to trap on this illegal access and abort our program.</p>
<p>This area of memory is set aside by the operating system because it indicates a programming error of not setting up an object or data structure properly.</p>
<p>When things go wrong, we don’t always get a crash. Only if it is Operating Environment policy then we get a crash.</p>
<h3 id="mac-address-example">MAC Address Example</h3>
<p>Consider the example of getting the MAC address of our iPhone. The Media Access Control (MAC) address is a unique code allocated to network cards to allow machines to talk to each other without duplication at the Data Link layer of the communication stack.</p>
<p>Prior to iOS 7, the MAC address was not considered a sensitive API. So requesting the MAC address using the <code>sysctl</code> API gave the real address. To see this in action, see the <code>icdab_sample</code> app <span class="citation" data-cites="icdabgithub">(“IOS Crash Dump Analysis Book Github Resources” 2018)</span>.</p>
<p>Unfortunately, the API was abused as a way of tracking the user - a privacy violation. Therefore, Apple introduced a policy from iOS 7 where they would return a fixed MAC address always.</p>
<p>Apple could have chosen to crash our app when any call to <code>sysctl</code> was made. However, <code>sysctl</code> is a general-purpose low-level call which can be used for other valid purposes. Therefore the policy set by iOS was to return a fixed MAC address <code>02:00:00:00:00:00</code> whenever that was requested.</p>
<h3 id="camera-example">Camera Example</h3>
<p>Now lets consider the case of taking a photo using the camera.</p>
<p>Introduced in iOS 10, when we want to access the Camera, a privacy sensitive feature, we need to define human readable text that is presented inside the system permission dialogue before access to the Camera is granted.</p>
<p>If we don’t define the text in our <code>Info.plist</code> for <code>NSCameraUsageDescription</code> we still see the following code evaluating true and then attempting to present the image picker.</p>
<pre><code>if UIImagePickerController.isSourceTypeAvailable(
UIImagePickerControllerSourceType.camera) {
// Use Xcode 9.4.1 to see it enter here
// Xcode 10.0 will skip over
let imagePicker = UIImagePickerController()
imagePicker.delegate = self
imagePicker.sourceType =
UIImagePickerControllerSourceType.camera
imagePicker.allowsEditing = false
self.present(imagePicker, animated: true, completion: nil)
}</code></pre>
<p>However when we run the code, via Xcode 9.4.1, we see a crash with a descriptive console message:</p>
<pre><code>2018-07-10 20:09:21.549897+0100 icdab_sample[1775:15601294]
[access] This app has crashed because it attempted to access
privacy-sensitive data without a usage description.
The app's Info.plist must contain an NSCameraUsageDescription
key with a string value explaining to the user how the app
uses this data.
Message from debugger: Terminated due to signal 9</code></pre>
<h3 id="lessons-learnt">Lessons Learnt</h3>
<p>Note the contrast here. In both cases there was a privacy sensitive API. However, in the camera case, Apple chose a policy of crashing our app instead of giving a warning, allowing a boilerplate standard explanation dialog, or returning a <code>false</code> value to indicate the source type was not available.</p>
<p>This seems like a harsh design choice. When Xcode 10.0 was introduced (it delivers the iOS 12 SDK) the behavior of the API changed. It returns <code>false</code> if the camera is not available due to a missing privacy string in the application <code>Info.plist</code></p>
<p>This underlies the point about there being two entities involved, the program and the operating environment (which includes its policies). Having correct source code does not guarantee crash free running. When we see a crash we need to think about the operating environment as much as the code itself.</p>
<h2 id="application-policies">Application policies</h2>
<p>The application we are writing can also request a crash. This is typically done via <code>assert</code> calls in our code. These calls ask the Operating Environment to terminate our app if any <code>assert</code> has failed. The Operating Environment then aborts our app. In the Crash Report we get a:</p>
<p><code>Exception Type: EXC_CRASH (SIGABRT)</code></p>
<p>to indicate it was the application that requested the crash in the first place.</p>
<h3 id="when-should-we-crash">When should we crash?</h3>
<p>We can apply similar standards as the Operating Environment for our crash policy.</p>
<p>If our code detects a data integrity issue, we could crash to prevent further data corruption.</p>
<h3 id="when-should-we-not-crash">When should we not crash?</h3>
<p>If the problems have resulted directly from some IO problem (file or network access for example) or some human input problem (such as a bad date value) then we should not crash.</p>
<p>It’s our job as the application developer to shield the lower level parts of the system from unpredictability present in the real world. Such problems are better dealt with by logging, error handling, user alerts, and IO retries.</p>
<h2 id="engineering-guidance">Engineering Guidance</h2>
<p>How should we guard against the privacy problems described above?</p>
<p>The thing to keep in mind is that any code that touches upon the policies the Operating Environment has guards for is a good candidate for automated testing.</p>
<p>In the <code>icdab_sample</code> project we have created Unit tests and UI tests.</p>
<p>Test cases always feel over-the-top when applied to trivial programs. But consider a large program that has an extensive <code>Info.plist</code> file. A new version of the app is called for so another <code>Info.plist</code> is created. Then keeping the privilege settings in sync between the different build targets becomes an issue. The UI test code shown here which merely launches the camera can catch such problems easily so has practical business value.</p>
<p>Similarly, if our app has a lot of low-level code and then is ported from iOS to tvOS, for example, how much of that OS-sensitive code is still applicable?</p>
<p>Unit testing a top level function comprehensively for different design concerns can pay off the effort invested in it before delving deeper and unit testing the underlying helper function calls in our code base. It’s a strategic play allowing us to get some confidence in our application and early feedback on problem areas when porting to other platforms within the Apple Ecosystem (and beyond).</p>
<h3 id="unit-testing-the-mac-address">Unit Testing the MAC Address</h3>
<p>The code to get the MAC address is not trivial. Therefore it merits some level of testing.</p>
<p>Here is a snippet from the Unit tests:</p>
<pre><code> func getFirstOctectAsInt(_ macAddress: String) -> Int {
let firstOctect = macAddress.split(separator: ":").first!
let firstOctectAsNumber = Int(String(firstOctect))!
return firstOctectAsNumber
}
func testMacAddressNotNil() {
let macAddress = MacAddress().getMacAddress()
XCTAssertNotNil(macAddress)
}
func testMacAddressIsNotRandom() {
let macAddressFirst = MacAddress().getMacAddress()
let macAddressSecond = MacAddress().getMacAddress()
XCTAssert(macAddressFirst == macAddressSecond)
}
func testMacAddressIsUnicast() {
let macAddress = MacAddress().getMacAddress()!
let firstOctect = getFirstOctectAsInt(macAddress)
XCTAssert(0 == (firstOctect & 1))
}
func testMacAddressIsGloballyUnique() {
let macAddress = MacAddress().getMacAddress()!
let firstOctect = getFirstOctectAsInt(macAddress)
XCTAssert(0 == (firstOctect & 2))
}</code></pre>
<p>In fact, the last test fails because the OS returns a local address.</p>
<h3 id="ui-testing-camera-access">UI Testing Camera access</h3>
<p>For testing camera access, we have written a simple UI test case which just presses the Take Photo button (by means of an accessibility identifier <code>takePhotoButton</code>)</p>
<pre><code>func testTakePhoto() {
let app = XCUIApplication()
app.buttons["takePhotoButton"].tap()
}</code></pre>
<p>This UI test code caused an immediate crash.</p>
<h1 id="tooling">Tooling</h1>
<h2 id="overview">Overview</h2>
<p>We have a rich set of tools available to assist crash dump analysis. When used properly they can save a huge amount of time.</p>
<p>Xcode provides much help out of the box. However, using and comprehending the information Xcode tools provide is daunting. In later chapters, we go through examples showing the use of such tools.</p>
<p>Additionally there are command line tools provided as standard in macOS. These are helpful when used in particular usage scenarios when we already know what we want to find out. We shall go through specific scenarios and show how the tools are used.</p>
<p>Next come software tools that help us reverse engineer programs. Sometimes we cannot get our program to work with a third party library. Aside from looking at Documentation or raising a Support Request, it’s possible to do some investigation ourselves using these tools.</p>
<h2 id="reverse-engineering">Reverse Engineering</h2>
<p>Reverse engineering is where an already built binary (such as an application, library, or helper process daemon), is studied to determine how it works. For a specific Object, we might want to find out:</p>
<ul>
<li>what are the lifecycles of the objects it is provided?</li>
<li>what checks does it do on objects?</li>
<li>what files or resources does it depend on?</li>
<li>why did it return a failure code?</li>
</ul>
<p>We generally do not want to know everything, only something specific to help build a hypothesis. Once we have a hypothesis, we will test it in relation to the crash dump we are dealing with.</p>
<p>How far should we go with reverse engineering, and how much money and time to invest in it is a good question. We offer the following recommendation:</p>
<ul>
<li>If we are just starting our application developer journey, or we have limited funds, then just stick with the standard Xcode tooling, macOS command line, and the open source class-dump tool.</li>
<li>If we are a professional application developer, we should strongly consider buying a commercial reverse engineering tool. The one that draws most attention is Hopper; it provides a lot of functionality offered by IDA Pro (a high end tool). It is well priced and can pay for itself in gained productivity even if only used a handful of times. We show how Hopper can be used in this book.</li>
<li>If we are a professional penetration tester, reverse engineer, or security researcher, then we will be probably wanting to invest in the top of the line software tool, IDA Pro. This tool costs thousands, but is often purchased as a company wide expense.</li>
</ul>
<h2 id="class-dump-tool">Class Dump Tool</h2>
<p>One of the great things about the Objective-C runtime is that it carries lots of rich program structure information in its built binaries. These allow the dynamic aspects of the language to work. In fact, its flexibility of dynamic dispatch is a source for many crashes.</p>
<p>We recommend installing the <code>class-dump</code> tool right away because we shall reference its usage in later chapters. See <span class="citation" data-cites="class-dump-tool">Nygard (2018)</span></p>
<p>The class dump tool allows us to look at what Objective C classes, methods and properties are present in a given program.</p>
<h2 id="third-party-crash-reporting">Third Party Crash Reporting</h2>
<p>The Apple Crash Reporter tool and supporting infrastructure in iTunes Connect is excellent but has some room for improvement.</p>
<p>A formidable piece of Open Source software, <code>plcrashreporter</code>, has been written by Landon Fuller, of Plausible Labs. <span class="citation" data-cites="plcrashreporter">(“Plausible Labs Crash Reporter” 2018)</span></p>
<p>The idea is to make our app handle all the possible signals and exceptions that can occur that would otherwise be unimplemented by the app and thus lead to the underlying Operating System to handle the crash.</p>
<p>With this solution, the crash data can be recorded, and then later communicated to a server of our own choice.</p>
<p>There are two benefits. Firstly, the crash handler can be fixed to handle edge cases not already handled by the Apple <code>ReportCrash</code> tool. Secondly, a more comprehensive server side solution can be employed.</p>
<p>For those wanting to explore, and understand, the Operating System, and low-level application code, <code>plcrashreporter</code> provides an excellent opportunity to study a well-engineered piece of system software.</p>
<p>When a company has many apps, many app variants, and has apps based on competitor platforms such as Android, a more powerful multi-platform solution is needed. Handling crash reports soon becomes a management problem. Which crash is the most serious? How many customers are affected? What are the metrics for quality and stability saying?</p>
<p>A number of commercial solutions are available, largely based upon the above Open Source project.</p>
<p>The Mobile Software Development field has grown into a big industry over the last few years. Many specialist companies serve App Developers as their customers. The field is very active in terms of mergers and acquisitions. Therefore, we cannot name the competitors in the Crash Reporting space in this book, as the list would be constantly changing.</p>
<p>A good place to start is the <code>rollout.io</code> blog posting that reviewed different players in the market. <span class="citation" data-cites="3rdpartycrashtools">(“IOS Crash Reporting Tools” 2017)</span></p>
<h1 id="xcode-built-in-help">Xcode Built-In Help</h1>
<p>Xcode provides significant help to developers in understanding and preventing crashes.</p>
<p>We think of Xcode in layers of sophistication, where at the lowest layer of sophistication Xcode directly tells us the common error it has seen with suggested corrections, up to the highest level were Xcode is telling the raw information, but we need Operating Systems knowledge to interpret the information ourselves.</p>
<p>We shall revisit Xcode configuration, setup and tooling many times. Nevertheless, let us first start off with the simple but high value assistance Xcode provides.</p>
<h2 id="xcode-diagnostic-settings">Xcode Diagnostic Settings</h2>
<p>By opening the project <code>icdab_sample</code> <span class="citation" data-cites="icdabgithub">(“IOS Crash Dump Analysis Book Github Resources” 2018)</span> and looking at the Scheme definition and then highlighting the Diagnostics tab we see the following options:</p>
<p><img src="screenshots/diagnostic_settings.png" /></p>
<h3 id="execution-methodology">Execution Methodology</h3>
<p>If we have a crash that is reproducible from our own developer environment and source code, then a methodology for finding is to switch on the appropriate diagnostic setting and then re-run our application.</p>
<p>As we become familiar with each diagnostic, we will know which option to switch on. We shall work through different scenarios so we understand when to use each. But when we are just starting out its worth just going through each one-by-one to get a feel for what is available. The basic approach is:</p>
<ol type="1">
<li>Write a Unit Test Case or UI Test Case that hits the problem.</li>
<li>Enable just one of the Diagnostic options from above starting with our best guess.</li>
<li>Run our tests.</li>
<li>Take note of any warning or console message from Xcode.</li>
<li>Repeat again but choosing a different diagnostic option if the problem is not understood.</li>
</ol>
<h3 id="analysis-methodology">Analysis Methodology</h3>
<p>Another complementary approach for analyzing and proactively avoiding crashes is to run the Code Analyzer. This is invoked using Command-Shift-B</p>
<p>In the sample app <code>icdab_sample</code> the Analyzer reports:</p>
<pre><code>/Users/faisalm/dev/icdab/source/icdab_sample/icdab_sample/
macAddress.m:22:12:
warning: Null pointer argument in call to string length function
assert(strlen(nullPointer) == 0);</code></pre>
<p>and conveniently marks up our source code</p>
<p><img src="screenshots/analyser_null.png" /></p>
<p>This can be switched on for whenever the project is built, either in shallow or deep mode according to how we feel the tradeoff should be done between slower more thorough analysis versus quicker build times with less analysis. It is in the Build Settings tab for the Xcode project file.</p>
<p><img src="screenshots/static_analyser_build.png" /></p>
<p>For a large project that has never had an Analysis report done, the output can be overwhelming. There will be some noise in the report but it generally does a good job. There will be duplication in the report because certain classes of error will repeat throughout the code.</p>
<p>If we are developing code using the Agile software methodology, then it is possible to frame the report as potential backlog items that can be worked upon during the time allocated for refactoring and maintenance.</p>
<p>In a large software project refactoring and maintenance should be around 20% of the work in a Sprint. Different viewpoints arise in this area. The author recommends doing such work alongside the normal development activities so long as no high-risk changes are amongst the work being done. For risky changes, leave that until after a major update of the app has been done. There is usually a lull where planning and strategy is developed following a release, which allows a convenient software engineering window to tackle such matters.</p>
<h4 id="ios-quickedit-app-case-study">iOS QuickEdit App Case Study</h4>
<p>Where the analyzer identifies potential crashes, from an economic point of view, it is a good investment to fix the problem. For example in the case of the QuickEdit iOS App, about 1 million lines of Objective-C, with 70 000 daily active users, the analyzer was run and found 13 clear crashing issues. We created one engineering story (“Fix top analyzer errors”). All 13 issues were fixed in the same day with testing taking two more days. Crashes are a top complaint from customers. Bugs found in the field typically are 20 times the effort and cost compared to those found in development. With a large population of users, potentially experiencing a severe crash bug, the cost of those 13 bugs could be 20 * 3 days = 60 days wasted effort.</p>
<p>QuickEdit due to its age only used manual reference counting in Objective-C. Despite this, it had a reliability of 99.5% based on app analytics. Only about 5% of engineering effort was needed to maintain this stability over time once the initial issues had been addressed.</p>
<h3 id="process-methodology">Process Methodology</h3>
<p>One way to drive out crashes from our app, particularly when we are in a large organization, is to factor it in our software development process.</p>
<p>When a developer proposes a code change in a pull request, get the developer to ensure no new analyzer warnings are introduced. We might consider the analyzer report as a robotically generated code review available to us for free. That is particularly helpful if we are working alone on a project with no peer to review our code.</p>
<p>When code is committed to a feature branch, have the automated tests run on it, with different diagnostics settings set. This can shake out problems automatically.</p>
<p>Before each release, schedule time to run some specific user cases under the memory profiler to look at memory usage or other key metrics. Record the highlights such as the peak memory usage as well as the profile file. Then when the following release is made we have a yardstick to see how things have changed both quantitatively and qualitatively.</p>
<h2 id="the-middle-road">The Middle Road</h2>
<p>Most software developers know what they “should” be doing; clean code, proper tests, code reviews, etc.</p>
<p>We recommend taking a measured approach. There is a time for hacking together a sample app to understand a concept. There is a time to write a prototype that just needs to prove a business use case. There is a time to write heavily trusted code used by many people.</p>
<p>We take the view that maximizing economic impact is the one that matters most because most developers are involved in professional software development. Alternatively, if we are working on non-commercial projects or hobby projects, the economic cost is really our personal free time which we will want to use most effectively.</p>
<p>We recommend:</p>
<ul>
<li>For Sample apps and concept exploration, just code the app.</li>
<li>For Prototype Development, just use the Execution Methodology when we hit problems.</li>
<li>For Individual Product Development, from the beginning, run the Analyzer automatically and informally incorporate it into our workflow when we see something important. From the beginning write tests but selectively where we get big impact.</li>
<li>For Team-based Product Development, add in the Process Methodology. Start becoming comprehensive with Testing.</li>
</ul>
<h1 id="hybrid-environments">Hybrid Environments</h1>
<p>We have seen that Xcode offers many automatic facilities for crash dump analysis and crash avoidance. However, these cannot get us all the answers we need. A complementary design oriented viewpoint is needed.</p>
<p>In this chapter, we shall look at a sample app <code>icdab_planets</code> that uses hybrid of programming languages and paradigms. It shows an example of why design insights must also be considered.</p>
<h2 id="program-structure">Program structure</h2>
<p>The <code>icdab_planets</code> sample app uses a mixture of C++, and Objective-C++. It relies on both STL data structures and traditional Objective-C data structures. <span class="citation" data-cites="icdabgithub">(“IOS Crash Dump Analysis Book Github Resources” 2018)</span></p>
<p>The model layer of the app is written in C++. The controller layer of the app is written in Objective-C++.</p>
<p>The purpose of the app is to tell us how many Pluto sized planets would fit inside Jupiter.</p>
<h2 id="paradigms">Paradigms</h2>
<p>Recall earlier we demonstrated that:</p>
<ul>
<li>Objective-C allows messaging to nil objects</li>
<li>C crashes upon NULL dereference</li>
</ul>
<p>Here we show how the C++ Standard Template Library has a back-fill strategy.</p>
<p>In the STL map abstraction (a Hash Table) when we query for an entry that does not exist, the STL will insert a new entry in the table for the key being queried, and then return that entry instead of returning an error or returning a nil.</p>
<h2 id="the-problem">The Problem</h2>
<p>In our sample app, which crashes upon launch, we have an assert that gets triggered.</p>
<pre><code>double pluto_volume = pluto.get_volume();
assert(pluto_volume != 0.0);
double plutos_to_fill_jupiter
= jupiter.get_volume() / pluto_volume;</code></pre>
<p>Enabling code Analysis will not find any issue or warning.</p>
<p>The assert is in place to avoid a division by zero. The fact that it is triggered is good because we know where to start debugging the problem.</p>
<p>Pluto’s volume is 0.0 because the code</p>
<pre><code>planet pluto = planet::get_planet_with_name("Pluto");</code></pre>
<p>returns a planet with zero diameter.</p>
<p>From the file <code>planet_data.hpp</code> we see the API that we rely upon is:</p>
<pre><code>static planet get_planet_with_name(string name);</code></pre>
<p>Therefore, whatever name we pass in, we should always get a <code>planet</code> in response; never a NULL.</p>
<p>The problem is that this API has not been thought deeply about. It has just been put together as a thin wrapper around the underlying abstractions that do the work.</p>
<p>We have</p>
<pre><code>planet planet::get_planet_with_name(string name) {
if (!database.loaded_data) {
database.load_data();
}
return database.planets[name];
}</code></pre>
<p>At first glance, it might be that the database failed to load data properly. In fact, the database is missing the entry for Pluto due to it no longer being considered a planet:</p>
<pre><code>void planet_database::load_data() {
planet planet_Mercury =
planet("Mercury", 4878.0, 57.9 * millionKm);
planets["Mercury"] = planet_Mercury;
planet planet_Venus =
planet("Venus", 12104, 108.2 * millionKm);
planets["Venus"] = planet_Venus;
planet planet_Earth =
planet("Earth", 12756, 149.6 * millionKm);
planets["Earth"] = planet_Earth;
planet planet_Mars =
planet("Mars", 6792, 227.9 * millionKm);
planets["Mars"] = planet_Mars;
planet planet_Jupiter =
planet("Jupiter", 142984, 778 * millionKm);
planets["Jupiter"] = planet_Jupiter;
planet planet_Saturn =
planet("Saturn", 120536, 1427 * millionKm);
planets["Saturn"] = planet_Saturn;
planet planet_Uranus =
planet("Uranus", 51118, 2870 * millionKm);
planets["Uranus"] = planet_Uranus;
planet planet_Neptune =
planet("Neptune", 49532, 4497 * millionKm);
planets["Neptune"] = planet_Neptune;
// No longer considered a planet but instead a dwarf planet
// planet planet_Pluto =
// planet("Pluto", 2370, 7375 * millionKm);
// planets["Pluto"] = planet_Pluto;
loaded_data = true;
}</code></pre>
<p>The problem indirectly is because <code>database.planets[name]</code> discovered that there was no entry for Pluto so created one via the no-arg constructor as this is the behaviour for STL map data structures.</p>
<pre><code>planet::planet() {
this->name = "";
this->diameter = 0.0;
this->distance_from_sun = 0.0;
}</code></pre>
<p>We see the constructor makes the diameter zero in this case.</p>
<h2 id="solutions">Solutions</h2>
<p>We see that the problem is not applying the paradigms of each framework and language properly and when we have a mixture of paradigms, those different assumptions get masked by each layer of abstraction.</p>
<p>In STL, we expect a <code>find</code> operation to be done, instead of the indexing operator. This allows the abstraction to flag the absence of the item being found.</p>
<p>In Objective-C we expect the lookup API to be a function which returns an index given the lookup name. In addition, the index would be <code>NSNotFound</code> when the operation failed.</p>
<p>In this code example, each layer of abstraction assumes the other side will re-map the edge case into an appropriate form.</p>
<h3 id="stl-solution">STL Solution</h3>
<p>We have a variant of the code which does things “properly” from an STL point of view. <span class="citation" data-cites="icdabgithub">(“IOS Crash Dump Analysis Book Github Resources” 2018)</span> It is <code>example/planets_stl</code>. On the consumer side, we have a helper method:</p>
<pre><code>- (BOOL)loadPlanetData {
auto pluto_by_find = planet::find_planet_named("Pluto");
auto jupiter_by_find = planet::find_planet_named("Jupiter");
if (planet::isEnd(jupiter_by_find) ||
planet::isEnd(pluto_by_find)) {
return NO;
}
pluto = pluto_by_find->second;
jupiter = jupiter_by_find->second;
return YES;
}</code></pre>
<p>This is hard to parse if we are mainly an Objective-C programmer. If the project is mainly a C++ project, with a thin platform-specific layer, then perhaps that is acceptable. If the code base just leverages C++ code from elsewhere, then a better solution is to confine the paradigms to their own files and apply the facade design pattern to give a version of the API following Objective-C paradigms on the platform-specific code side.</p>
<p>Then Objective-C++ can be dispensed with in the ViewController code; it can be made an Objective-C file instead.</p>
<h3 id="facade-solution">Facade Solution</h3>
<p>Here is a facade implementation <code>example/facade_planets</code> that overcomes the mixing of paradigms problem.</p>
<p>The facade is:</p>
<pre><code>@implementation PlanetModel
- (id)init {
self = [super init];
NSString *testSupportAddPluto =
[[[NSProcessInfo processInfo] environment]
objectForKey:@"AddPluto"];
if ([testSupportAddPluto isEqualToString:@"YES"]) {
planet::add_planet(
planet("Pluto", 2370, 7375 * millionKm));
}
if (self) {
_planetDict = [[NSMutableDictionary alloc] init];
auto pluto_by_find =
planet::find_planet_named("Pluto");
auto jupiter_by_find =
planet::find_planet_named("Jupiter");
if (planet::isEnd(jupiter_by_find) ||
planet::isEnd(pluto_by_find)) {
return nil;
}
auto pluto = pluto_by_find->second;
auto jupiter = jupiter_by_find->second;
PlanetInfo *plutoPlanet = [[PlanetInfo alloc] init];
plutoPlanet.diameter = pluto.get_diameter();
plutoPlanet.distanceFromSun =
pluto.get_distance_from_sun();
plutoPlanet.volume = pluto.get_volume();
assert (plutoPlanet.volume != 0.0);
[_planetDict setObject:plutoPlanet forKey:@"Pluto"];
PlanetInfo *jupiterPlanet = [[PlanetInfo alloc] init];
jupiterPlanet.diameter = jupiter.get_diameter();
jupiterPlanet.distanceFromSun =
jupiter.get_distance_from_sun();
jupiterPlanet.volume = jupiter.get_volume();
assert (jupiterPlanet.volume != 0.0);
[_planetDict setObject:jupiterPlanet forKey:@"Jupiter"];
}
return self;
}
@end</code></pre>
<p>The consumer then becomes a purely Objective-C class:</p>
<pre><code>- (void)viewDidLoad {
[super viewDidLoad];
self.planetModel = [[PlanetModel alloc] init];
if (self.planetModel == nil) {
return;
}
double pluto_diameter =
self.planetModel.planetDict[@"Pluto"].diameter;
double jupiter_diameter =
self.planetModel.planetDict[@"Jupiter"].diameter;
double plutoVolume =
self.planetModel.planetDict[@"Pluto"].volume;
double jupiterVolume =
self.planetModel.planetDict[@"Jupiter"].volume;
double plutosInJupiter = jupiterVolume/plutoVolume;
self.plutosInJupiterLabelOutlet.text =
[NSString stringWithFormat:
@"Number of Plutos that fit inside Jupiter = %f",
plutosInJupiter];
self.jupiterLabelOutlet.text =
[NSString stringWithFormat:
@"Diameter of Jupiter (km) = %f",
jupiter_diameter];
self.plutoLabelOutlet.text =
[NSString stringWithFormat:
@"Diameter of Pluto (km) = %f",
pluto_diameter];
}</code></pre>
<h2 id="lessons-learnt-1">Lessons Learnt</h2>
<p>The lesson here is that crashes can arise from special case handling. Since different languages and frameworks deal with special cases in their own idiomatic manner, it is safer to separate out our code and use a Facade if possible to keep each paradigm cleanly separated.</p>
<h1 id="symbolification">Symbolification</h1>
<p>This chapter explains crash dump symbolification. Symbolification is the process of mapping machine addresses into symbolic addresses meaningful to the programmer possessing the source code. Instead of seeing machine addresses, we want to see function names (plus any offset),</p>
<p>We use the <code>icdab_planets</code> sample app to demonstrate a crash. <span class="citation" data-cites="icdabgithub">(“IOS Crash Dump Analysis Book Github Resources” 2018)</span></p>
<p>When dealing with real world crashes, a number of different entities are involved. These can be the end user device, the settings allowing the Crash Report to be sent back to Apple, the symbols held by Apple and our local development environment setup to mirror such a configuration.</p>
<p>In order to understand how things all fit together it is best to start from first principles and do the data conversion tasks ourselves so if we have to diagnose symbolification issues, we have some experience with the technologies at hand.</p>
<h2 id="build-process">Build Process</h2>
<p>Normally when we develop an app, we are deploying the Debug version of our app onto our device. When we are deploying our app for testers, app review, or app store release, we are deploying the Release version of our app.</p>
<p>By default for Release builds, debug information from the <code>.o</code> object files is placed into a separate directory structure. It is called <code>our_app_name.DSYM</code></p>
<p>The debugger can use debugging information when it sees a crash to help us understand where the program has gone wrong.</p>
<p>When a user sees our program crash, there is no debugger. Instead, a crash report is generated. This comprises the machine addresses where the problem was seen. Symbolification can convert these addresses into meaningful source code references.</p>
<p>In order for symbolification to occur, appropriate DSYM files must exist.</p>
<p>Xcode is by default setup so that only DSYM files are generated for Release builds, and not for Debug builds.</p>
<h2 id="build-settings">Build Settings</h2>
<p>From Xcode, in our build settings, searching for “Debug Information Format” we see the following settings:</p>
<table>
<colgroup>
<col style="width: 33%" />
<col style="width: 33%" />
<col style="width: 33%" />
</colgroup>
<thead>
<tr class="header">
<th>Setting</th>
<th>Meaning</th>
<th>Usually set for target</th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td>DWARF</td>
<td>Debugging information is in <code>.o</code> files only</td>
<td>Debug</td>
</tr>
<tr class="even">
<td>DWARF with dSYM File</td>
<td>As before but also collates the debug information into a DSYM file</td>
<td>Release</td>
</tr>
</tbody>
</table>
<p>In the default setup, if we run our debug binary on our device, launching it from the app icon itself then if it were to crash we would not have any symbols in the Crash Report. This confuses many people.</p>
<p>The problem is that the UUID of the binary and the DSYM need to match.</p>
<p>To avoid this problem, the sample app <code>icdab_planets</code> has been configured to have <code>DWARF with dSYM File</code> set for both debug and release targets. Then symbolification will work, because there will be a matching DSYM on our Mac.</p>
<h2 id="observing-a-local-crash">Observing a local crash</h2>
<p>The <code>icdab_planets</code> program is designed to crash upon launch due to an assertion.</p>
<p>If the <code>DWARF with dSYM File</code> setting had not been made, we would get a partially symbolicated crash.</p>
<p>The Crash Report, seen from <em>Windows->Devices and Simulators->View Device Logs</em>, would look like this (truncated for ease of demonstration)</p>
<pre><code>Thread 0 Crashed:
0 libsystem_kernel.dylib
0x0000000183a012ec __pthread_kill + 8
1 libsystem_pthread.dylib
0x0000000183ba2288 pthread_kill$VARIANT$mp + 376
2 libsystem_c.dylib
0x000000018396fd0c abort + 140
3 libsystem_c.dylib
0x0000000183944000 basename_r + 0
4 icdab_planets
0x00000001008e45bc 0x1008e0000 + 17852
5 UIKit
0x000000018db56ee0
-[UIViewController loadViewIfRequired] + 1020
Binary Images:
0x1008e0000 - 0x1008ebfff icdab_planets arm64
<9ff56cfacd66354ea85ff5973137f011>
/var/containers/Bundle/Application/
BEF249D9-1520-40F7-93F4-8B99D913A4AC/
icdab_planets.app/icdab_planets</code></pre>
<p>However, with the setting in place, a crash would instead be reported as:</p>
<pre><code>Thread 0 Crashed:
0 libsystem_kernel.dylib
0x0000000183a012ec __pthread_kill + 8
1 libsystem_pthread.dylib
0x0000000183ba2288
pthread_kill$VARIANT$mp + 376
2 libsystem_c.dylib
0x000000018396fd0c abort + 140
3 libsystem_c.dylib
0x0000000183944000 basename_r + 0
4 icdab_planets
0x0000000104e145bc
-[PlanetViewController viewDidLoad] + 17852
(PlanetViewController.mm:33)
5 UIKit
0x000000018db56ee0
-[UIViewController loadViewIfRequired] + 1020</code></pre>
<p>Lines 0, 1, 2, 5 are the same in both cases because our developer environment will have the symbols for the iOS release under test. In the second case, Xcode will look up the DSYM file to clarify line 4. It tells us this is line 33 in file PlanetViewController.mm. This is:</p>
<pre><code>assert(pluto_volume != 0.0);</code></pre>
<h2 id="dsym-structure">DSYM structure</h2>
<p>The DSYM file is strictly speaking a directory hierarchy:</p>
<pre><code>icdab_planets.app.dSYM
icdab_planets.app.dSYM/Contents
icdab_planets.app.dSYM/Contents/Resources
icdab_planets.app.dSYM/Contents/Resources/DWARF
icdab_planets.app.dSYM/Contents/Resources/DWARF/icdab_planets
icdab_planets.app.dSYM/Contents/Info.plist</code></pre>
<p>It is just the DWARF data normally put into the intermediate <code>.o</code> files but copied into a separate file.</p>
<p>From looking at our build log, we can see how the DSYM was generated. It is effectively just <code>dsymutil path_to_app_binary -o output_symbols_dir.dSYM</code></p>
<h2 id="manual-symbolification">Manual Symbolification</h2>
<p>In order to help us get comfortable with crash dump reports, we can demonstrate how the symbolification actually works. In the first crash dump, we want to understand:</p>
<pre><code>4 icdab_planets
0x00000001008e45bc 0x1008e0000 + 17852</code></pre>
<p>If we knew accurately the version of our code at the time of the crash, we can recompile our program, but with the DSYM setting switched on, and then get a DSYM file after the original crash. It should line up almost exactly.</p>
<p>The crash dump program tells us where the program was loaded, in memory, at the time of the problem. That tells us the master base offset from which all other address (TEXT) locations are relative to.</p>
<p>At the bottom of the crash dump, we have line <code>0x1008e0000 - 0x1008ebfff icdab_planets</code> Therefore, the icdab_planets binary starts at location <code>0x1008e0000</code></p>
<p>Running the lookup command <code>atos</code> symbolicates the line of interest:</p>
<pre><code># atos -arch arm64 -o
./icdab_planets.app.dSYM/Contents/Resources/DWARF/
icdab_planets -l 0x1008e0000 0x00000001008e45bc
-[PlanetViewController viewDidLoad] (in icdab_planets)
(PlanetViewController.mm:33)</code></pre>
<p>The Crash Reporter tool fundamentally just uses <code>atos</code> to symbolicate the Crash Report, as well as providing other system related information.</p>
<p>Symbolification is described further by an Apple Technote in case we want to get into it in more detail. <span class="citation" data-cites="tn2123">(“CrashReport Technote 2123” 2004)</span></p>
<h2 id="reverse-engineering-approach">Reverse Engineering Approach</h2>
<p>In the above example we have the source code, and symbols, for the crash dump so can do Symbolification.</p>
<p>Sometimes we may have included a third party binary framework in our project for which we do not have the source code. It is good practice for the vendor to supply symbol information for their framework to allow crash dump analysis. When symbol information is not available, it is still possible to make progress by applying some reverse engineering.</p>
<p>When working with third parties there is typically a much larger turnaround time for diagnostics and troubleshooting. We find that well written and specific bug reports can speed up things a lot. The following approach can help provide the kind of specific information needed.</p>
<p>We shall demonstrate our approach using the Hopper tool mentioned in the Tooling chapter.</p>
<p>Launching hopper, we choose <em>File->Read Executable to Disassemble</em>. The binary in our case is <code>examples/assert_crash_ios/icdab_planets</code></p>
<p>We need to “rebase” our disassembly so the addresses it shows mirror those of the program when it crashed. We choose <em>Modify->Change File Base Address</em>. As before, we supply <code>0x1008e0000</code>.</p>
<p>Now we can visit the code that crashed. The address <code>0x00000001008e45bc</code> is actually the address the device would <em>return</em> to after performing the function call in the stack trace. Nevertheless, it puts us in the right part of the file. We choose <em>Navigate->Go To Address or Symbol</em> and supply <code>0x00000001008e45bc</code></p>
<p>The overall view we see is</p>
<p><img src="screenshots/hopperAddressView.png" /></p>
<p>Zooming in on the code line, we have</p>
<p><img src="screenshots/hopperPlanetAbort.png" /></p>
<p>This indeed shows the return address for the assert method. Further up, we see the test for Pluto’s volume being non-zero. This is just a very basic Hopper example. We shall revisit Hopper later to demonstrate its most interesting feature - that of being able to generate pseudocode from assembly code. This lowers the mental load of comprehending crashes. Most developers rarely look at assembly code nowadays so this feature is worth the cost of the software itself!</p>
<p>Now at least for the current problem, we could formulate a bug report that said the code was crashing because Pluto’s volume was zero. That may be enough to unlock the problem from the framework vendor’s point of view.</p>
<p>In a more complex case, imagine we were using an image conversion library that was crashing.<br />
There can be many pixel formats for images. An <code>assert</code> might lead us to notice it was the particular format that was asserting. Therefore, we could just try a different pixel format.</p>
<p>Another example would be a security library. Security code often gives back generic error codes, not specific fault codes to allow for future code enhancement and avoid leaking internal details (a security risk). A crash dump in a security library might point out exactly the kind of security issue, and help us correct some data structure passed into the library much earlier on.</p>
<h1 id="the-crash-report">The Crash Report</h1>
<p>In this chapter, we get into the details of what comprises a Crash Report. Our main focus is the iOS Crash Report. We also cover the macOS Crash Report, which carries a slightly different structure but serves the same purpose.</p>
<p>Note, it is possible for an app to install crash handlers from third parties, either to get enhanced crash reporting diagnostics, or to link application crashes to a web-based service for managing Crash Reports across a potentially large population of users. In this chapter we assume the app has not done this, and therefore the Apple CrashReport tool comes into play.</p>
<p>When a crash occurs the <code>ReportCrash</code> program extracts information from the crashing process from the Operating System. The result is a text file with a <code>.crash</code> extension.</p>
<p>When symbol information is available, Xcode will symbolicate the Crash Report to show symbolic names instead of machine addresses. This improves the comprehensibility of the report.</p>
<p>Apple has produced a detailed document explaining the anatomy of a crash dump. <span class="citation" data-cites="tn2151">(“Apple Crash Dump Technote 2151” 2018)</span></p>
<h2 id="system-diagnostics">System Diagnostics</h2>
<p>Crash Reports are just one part of a much bigger diagnostic reporting story.</p>
<p>Ordinarily as application developers, we don’t need to look much further. However, if our problems are potentially triggered by an unexplained series of events or a more complex system interaction with hardware or Apple provided system services, then not only do we need to look at our Crash Reports, we need to study the system diagnostics.</p>
<h3 id="extracting-system-diagnostic-information">Extracting System Diagnostic Information</h3>
<p>When understanding the environment that gave rise to our crash, we may need to install Mobile Device Management Profiles (to switch on certain debugging subsystems), or create virtual network interfaces (for network sniffing). Apple provides a great web page covering each scenario. <span class="citation" data-cites="apple-sysdiag">(“Diagnostic Profiles and Logs” 2018)</span></p>
<p>On iOS, the basic idea is that we install a profile, which alters our device to produce more logging, and then we reproduce the crash (or get the customer to do that). Then we press a special key sequence on the device (for example, both volume buttons and the side button). The system vibrates briefly to indicate it is running a program, <code>sysdiagnose</code>, which extracts many log files. Then we use iTunes to synchronize our device to retrieve the resultant <code>sysdiagnose_date_name.tar.gz</code> file. Inside this archive file are many system and subsystem logs, and we can see when crashes occur and the context that gave rise to them.</p>
<p>An equivalent approach is available on macOS as well.</p>
<h2 id="guided-tour-of-an-ios-crash-report">Guided tour of an iOS Crash Report</h2>
<p>Here we go through each section of an iOS Crash Report and explain the fields. <span class="citation" data-cites="tn2151">(“Apple Crash Dump Technote 2151” 2018)</span></p>
<p>tvOS and watchOS may be just considered subsets of iOS for our purposes and have similar Crash Reports.</p>
<p>Note here we use the term “iOS Crash Report” to mean a Crash Report that came from a physical target device. After a crash, apps are often debugged on the Simulator. The exception code may be different in that case because the Simulator uses different methodology to cause the app to stop under the debugger.</p>
<h3 id="ios-crash-report-header-section">iOS Crash Report Header Section</h3>
<p>A Crash Report starts with the following header:</p>
<pre><code>Incident Identifier: E030D9E4-32B5-4C11-8B39-C12045CABE26
CrashReporter Key: b544a32d592996e0efdd7f5eaafd1f4164a2e13c
Hardware Model: iPad6,3
Process: icdab_planets [2726]
Path: /private/var/containers/Bundle/Application/
BEF249D9-1520-40F7-93F4-8B99D913A4AC/
icdab_planets.app/icdab_planets
Identifier: www.perivalebluebell.icdab-planets
Version: 1 (1.0)
Code Type: ARM-64 (Native)
Role: Foreground
Parent Process: launchd [1]
Coalition: www.perivalebluebell.icdab-planets [1935]</code></pre>
<p>These items are explained by the following table:</p>
<table>
<colgroup>
<col style="width: 50%" />
<col style="width: 50%" />
</colgroup>
<thead>
<tr class="header">
<th>Entry</th>
<th>Meaning</th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td>Incident Identifier</td>
<td>Unique report number of crash</td>
</tr>
<tr class="even">
<td>CrashReporter Key</td>
<td>Unique identifier for the device that crashed</td>
</tr>
<tr class="odd">
<td>Hardware Model</td>
<td>Apple Hardware Model <span class="citation" data-cites="ios-devices">(“List of iOS Devices” 2018)</span></td>
</tr>
<tr class="even">
<td>Process</td>
<td>Process name (number) that crashed</td>
</tr>
<tr class="odd">
<td>Path</td>
<td>Full pathname of crashing program on the device file system</td>
</tr>
<tr class="even">
<td>Identifier</td>
<td>Bundle identifier from <code>Info.plist</code></td>
</tr>
<tr class="odd">
<td>Version</td>
<td>CFBundleVersion; also CFBundleVersionString in brackets</td>
</tr>
<tr class="even">
<td>Code Type</td>
<td>Target architecture of the process that crashed</td>
</tr>
<tr class="odd">
<td>Role</td>
<td>The process <code>task_role</code>. An indicator if we were in the background, foreground, or was a console app. Mainly affects the scheduling priority of the process.</td>
</tr>
<tr class="even">
<td>Parent Process</td>
<td>Parent of the crashing process. <code>launchd</code> is a process launcher and is often the parent.</td>
</tr>
<tr class="odd">
<td>Coalition</td>
<td>Tasks are grouped into coalitions so they can pool together their consumption of resources <span class="citation" data-cites="resource-management">(“Resource Management” 2015)</span></td>
</tr>
</tbody>
</table>
<p>The first thing to look at is the version. Typically, if we are a small team or an individual, we will not have the resources to diagnose crashes in older versions of our app, so the first thing might be to get the customer to install the latest version.</p>
<p>If we have many crashes then a pattern may emerge. It could be one customer (common CrashReporter key seen), or many customers (different CrashReporter keys seen). This may affect how we rank the priority of the crash.</p>
<p>The hardware model could be interesting. Is it iPad only devices, or iPhone only, or both? Maybe our code has less testing or unique code paths for a given platform. The hardware model might indicate an older device, which we have not tested on.</p>
<p>Whether the app crashed in the Foreground or Background (the Role) is interesting because most applications are not tested whilst they are in the background. For example, we might receive a phone call, or have task switched between apps.</p>
<p>The Code Type (target architecture) is now mostly 64-bit ARM. However, we might see ARM being reported - the original 32-bit ARM.</p>
<h3 id="ios-crash-report-date-and-version-section">iOS Crash Report Date and Version Section</h3>
<p>A Crash Report will continue with date and version information:</p>
<pre><code>Date/Time: 2018-07-16 10:15:31.4746 +0100
Launch Time: 2018-07-16 10:15:31.3763 +0100
OS Version: iPhone OS 11.3 (15E216)
Baseband Version: n/a
Report Version: 104</code></pre>
<p>These items are explained by the following table:</p>
<table>
<colgroup>
<col style="width: 50%" />
<col style="width: 50%" />
</colgroup>
<thead>
<tr class="header">
<th>Entry</th>
<th>Meaning</th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td>Date/Time</td>
<td>When the crash occurred</td>
</tr>
<tr class="even">
<td>Launch Time</td>
<td>When the process was originally launched before crashing</td>
</tr>
<tr class="odd">
<td>OS Version</td>
<td>Operating System Version (Build number). <span class="citation" data-cites="ios-versions">(“IOS Version History” 2018)</span></td>
</tr>
<tr class="even">
<td>Baseband Version</td>
<td>Version number of the firmware of the cellular modem (used for phone calls) or <code>n/a</code> if the device has no cellular modem (most iPads, iPod Touch, etc.)</td>