Input/Output error

[mmirg]

I'm not sure if this is specific to my particular hardware, but applying these power patches to the kernel on my Precision 5510 (workstation analogue of the XPS 15 9550) seems to result in instability that ultimately leads to the OS complaining of input/output errors as if there was something wrong with the underlying storage hardware. I have not noticed this issue when using Linux 4.7.3 with its older NVMe codebase. Is this something that has been noticed on any other hardware?

[damige]

https://bbs.archlinux.org/viewtopic.php?id=205147&p=40 post 999 might be related. It seems to refer to the vanilla 4.8 kernel.
Have you tried the vanilla 4.8 kernel,?like you said there are other changes in the nvme codebase between 4.7.x and 4.8.

I have had a quick look at backporting the nvme patches to 4.7.x. Its not trivial.

[mmirg]

I've tried with the vanilla 4.8 kernel and get no such errors. I get the errors with the v4 nvme patches from Andy Lutomirski with what I believe is a Samsung 950 Pro (Device: pci 0xa802). When I ordered this laptop, this drive was exclusive to the workstation model and not available on the XPS 15 9550, I'm not sure if it's an incompatibility between the specific hardware and the patches. Either way, I suspect I should probably report this upstream to the LKML as well.

[damige]

Just on the off-chance that you ran into the bug in 4.8:(http://lkml.iu.edu/hypermail/linux/kernel/1610.0/00878.html) i have updated it to 4.8.1.

If this does not fix it (chances would be slim as you tried vanilla), reporting upstream is the way to go. Please keep us updated if you do.

[mmirg]

I've reported this bug upstream as it is consistent for me across all versions of 4.8.x that have been released so far. I have a Dell engineer in the loop as well in case there is something specific to the platform that is contributing to this issue. Thanks for your quick feedback on it and I'll update here if anything gets resolved (or at least clarified.)

There was a similar bug report made here: NVMe device suddenly unavailable that kind of fizzled away without resolution but it doesn't appear to be related specifically to the NVMe APST patches so it could be that I'm actually seeing some other issue that is coincidental to applying the patch.

[damige]

Great to hear! i will leave this issue open so we can receive your feedback in it. Thanks!

[mmirg]

Quick update on here, we've debugged this problem to where the upstream kernel developers have identified a possible solution. Andy is working on a new version of the patch. I can copy you on the mail thread instead of sending updates here if it's a more convenient way to track the issue.

[mmirg]

There's an updated experimental patch set for this that attempts to fix the bug that I reported upstream. It's available at 20160512-nvme-test. By default, applying this patch set (which includes eight patches if you go up the tree from 1a075417a8c9 nvme/scsi: Remove START STOP emulation to aab102ebbaae dev_pm_qos:
Improve sysfs pm_qos_latency_tolerance validation) disables APST using a quirk for the Samsung 951 (which is the drive that I have). If you wish to enable the APST functionality for that drive you can revert 452aefdcb67a nvme: Add a quirk to disable APST on a buggy Samsung device. I'm currently testing the patch now, should it work properly, I imagine we would start to see a cleaner patch set soon. As it is right now, it includes a fair bit of debugging output to logs. While there is a chance the new patches might work when APST is enabled, the author still feels that this is unlikely and would require a bit more work.

[tlvince]

Thanks for the updates on this. Would this also affect the PM951? The "quirk table" patch doesn't specifically include it, but I wonder if there any lasting consequences before I go about trying it 😄

[damige]

I have not heard of any lasting consequences. The patches included in my 4.10 only differ slightly from what you will find in mainline 4.11rcX.

[tlvince]

Thanks. I'm running the APST-full patches against 4.10.3 and am seeing ~1W reduction at idle 👍

[mmirg]

To the best of my knowledge, this is an issue that's specific to the model listed in the quirk table and possibly even the Dell platform that I encountered the bug on. The bug has finally been confirmed and replicated by Dell and is being formally investigated by Samsung in Korea (as of this morning) on the previous generation XPS 15 9550/Precision 5510 platform. I'm in the email chain and I'll keep this issue updated with any relevant information.

[damige]

Great thanks ! i will reopen this so that people know this is ongoing.

[mmirg]

With this patch now staged for inclusion in 4.11, more people are experiencing this bug. Two upstream reports can be tracked here:

Bug 194921 - Kernel oopses/panics after controller gets reset
Bug 195039 - Samsung PM951 NVMe sudden controller death

I guess this issue project will be moot once the patches are upstreamed, but for the time being, I thought I'd report these here for those affected to be able to keep tabs on the relevant bugs.

[lllars]

I think I'm running into this issue with a PM961. Hard to say for sure because it has only happened 3 times so far (the first time about 3 weeks ago). I'm definitely getting input/output errors each time, but I haven't managed to capture any error logs yet, and it occurs so infrequently it would be hard to test whether a change to APST settings makes any difference.

I do see that someone else with a PM961 seems to be having the problem. See the end of the "NVMe sudden controller death" bug.

Please let me know if there's any other info I can provide, or if there's a more appropriate place to post.

$ uname -a
Linux tank 4.12.8-2-ARCH #1 SMP PREEMPT Fri Aug 18 14:08:02 UTC 2017 x86_64 GNU/Linux

$ modinfo nvme_core
filename:       /lib/modules/4.12.8-2-ARCH/kernel/drivers/nvme/host/nvme-core.ko.gz
version:        1.0
license:        GPL
srcversion:     AADA53B606A46DD4B54F7C9
depends:        
intree:         Y
vermagic:       4.12.8-2-ARCH SMP preempt mod_unload modversions 
parm:           admin_timeout:timeout in seconds for admin commands (byte)
parm:           io_timeout:timeout in seconds for I/O (byte)
parm:           shutdown_timeout:timeout in seconds for controller shutdown (byte)
parm:           max_retries:max number of retries a command may have (byte)
parm:           nvme_char_major:int
parm:           default_ps_max_latency_us:max power saving latency for new devices; use PM QOS to change per device (ulong)
parm:           force_apst:allow APST for newly enumerated devices even if quirked off (bool)

$ sudo nvme id-ctrl /dev/nvme0
[sudo] password for lars: 
NVME Identify Controller:
vid     : 0x144d
ssvid   : 0x144d
sn      : S2U3NX0J103036      
mn      : SAMSUNG MZVLW1T0HMLH-00000              
fr      : CXY7301Q
rab     : 2
ieee    : 002538
cmic    : 0
mdts    : 0
cntlid  : 2
ver     : 10200
rtd3r   : 186a0
rtd3e   : 4c4b40
oaes    : 0
oacs    : 0x17
acl     : 7
aerl    : 3
frmw    : 0x16
lpa     : 0x3
elpe    : 63
npss    : 4
avscc   : 0x1
apsta   : 0x1
wctemp  : 341
cctemp  : 344
mtfa    : 0
hmpre   : 0
hmmin   : 0
tnvmcap : 1024209543168
unvmcap : 0
rpmbs   : 0
sqes    : 0x66
cqes    : 0x44
nn      : 1
oncs    : 0x1f
fuses   : 0
fna     : 0
vwc     : 0x1
awun    : 255
awupf   : 0
nvscc   : 1
acwu    : 0
sgls    : 0
subnqn  : 
ps    0 : mp:7.60W operational enlat:0 exlat:0 rrt:0 rrl:0
          rwt:0 rwl:0 idle_power:- active_power:-
ps    1 : mp:6.00W operational enlat:0 exlat:0 rrt:1 rrl:1
          rwt:1 rwl:1 idle_power:- active_power:-
ps    2 : mp:5.10W operational enlat:0 exlat:0 rrt:2 rrl:2
          rwt:2 rwl:2 idle_power:- active_power:-
ps    3 : mp:0.0400W non-operational enlat:210 exlat:1500 rrt:3 rrl:3
          rwt:3 rwl:3 idle_power:- active_power:-
ps    4 : mp:0.0050W non-operational enlat:2200 exlat:6000 rrt:4 rrl:4
          rwt:4 rwl:4 idle_power:- active_power:-`