[FSL P50x0] KVM HV doesn't work anymore

Hi All,

I bisected today [1] and the bisecting itself was OK but the reverting
of the bad commit doesn't solve the issue. Do you have an idea which
commit could be resposible for this issue? Maybe the bisecting wasn't
successful. I will look in the kernel git log. Maybe there is a commit
that affected KVM HV on FSL P50x0 machines.

Thanks,
Christian

[1] https://forum.hyperion-entertainment.com/viewtopic.php?p=53209#p53209

Post by Christian Zigotzky
Hi All,
The RC1 of kernel 5.13 doesn't boot in a virtual e5500 QEMU machine
with KVM HV anymore. I see in the serial console that the uImage
qemu-system-ppc64 -M ppce500 -cpu e5500 -enable-kvm -m 1024 -kernel
uImage-5.13 -drive
format=raw,file=MintPPC32-X5000.img,index=0,if=virtio -netdev
user,id=mynet0 -device e1000,netdev=mynet0 -append "rw root=/dev/vda"
-device virtio-vga -device virtio-mouse-pci -device
virtio-keyboard-pci -device pci-ohci,id=newusb -device
usb-audio,bus=newusb.0 -smp 4
The kernel boots without KVM HV.
Have you already tested KVM HV with the kernel 5.13?
Thanks,
Christian

Christophe Leroy

2021-05-15 10:08:10 UTC

Post by Christian Zigotzky
Hi All,
I bisected today [1] and the bisecting itself was OK but the reverting of the bad commit doesn't
solve the issue. Do you have an idea which commit could be resposible for this issue? Maybe the
bisecting wasn't successful. I will look in the kernel git log. Maybe there is a commit that
affected KVM HV on FSL P50x0 machines.

If the uImage doesn't load, it may be because of the size of uImage.

See https://github.com/linuxppc/issues/issues/208

Is there a significant size difference with and without KVM HV ?

Maybe you can try to remove another option to reduce the size of the uImage.

Or if you are using gzipped uImage you can try with an lzma uImage. You can find a way to get an
lzma uImage here: https://github.com/linuxppc/issues/issues/208#issuecomment-477479951

Christophe

Post by Christian Zigotzky
Thanks,
Christian
[1] https://forum.hyperion-entertainment.com/viewtopic.php?p=53209#p53209

Post by Christian Zigotzky
Hi All,
The RC1 of kernel 5.13 doesn't boot in a virtual e5500 QEMU machine with KVM HV anymore. I see in
qemu-system-ppc64 -M ppce500 -cpu e5500 -enable-kvm -m 1024 -kernel uImage-5.13 -drive
format=raw,file=MintPPC32-X5000.img,index=0,if=virtio -netdev user,id=mynet0 -device
e1000,netdev=mynet0 -append "rw root=/dev/vda" -device virtio-vga -device virtio-mouse-pci -device
virtio-keyboard-pci -device pci-ohci,id=newusb -device usb-audio,bus=newusb.0 -smp 4
The kernel boots without KVM HV.
Have you already tested KVM HV with the kernel 5.13?
Thanks,
Christian

Christian Zigotzky

2021-05-15 13:46:06 UTC

Post by Christian Zigotzky
Hi All,
I bisected today [1] and the bisecting itself was OK but the
reverting of the bad commit doesn't solve the issue. Do you have an
idea which commit could be resposible for this issue? Maybe the
bisecting wasn't successful. I will look in the kernel git log. Maybe
there is a commit that affected KVM HV on FSL P50x0 machines.

I tried it but it doesn't solve the issue. The uImage works without KVM
HV in a virtual e5500 QEMU machine.

-Christian

Post by Christophe Leroy
Or if you are using gzipped uImage you can try with an lzma uImage.
https://github.com/linuxppc/issues/issues/208#issuecomment-477479951
Christophe

Post by Christian Zigotzky
Thanks,
Christian
[1]
https://forum.hyperion-entertainment.com/viewtopic.php?p=53209#p53209

Post by Christian Zigotzky
Hi All,
The RC1 of kernel 5.13 doesn't boot in a virtual e5500 QEMU machine
with KVM HV anymore. I see in the serial console that the uImage
qemu-system-ppc64 -M ppce500 -cpu e5500 -enable-kvm -m 1024 -kernel uImage-5.13 -drive
format=raw,file=MintPPC32-X5000.img,index=0,if=virtio -netdev
user,id=mynet0 -device e1000,netdev=mynet0 -append "rw
root=/dev/vda" -device virtio-vga -device virtio-mouse-pci -device
virtio-keyboard-pci -device pci-ohci,id=newusb -device
usb-audio,bus=newusb.0 -smp 4
The kernel boots without KVM HV.
Have you already tested KVM HV with the kernel 5.13?
Thanks,
Christian

Nicholas Piggin

2021-05-17 07:42:23 UTC

Post by Christian Zigotzky
Hi All,
I bisected today [1] and the bisecting itself was OK but the
reverting of the bad commit doesn't solve the issue. Do you have an
idea which commit could be resposible for this issue? Maybe the
bisecting wasn't successful. I will look in the kernel git log. Maybe
there is a commit that affected KVM HV on FSL P50x0 machines.

I tried it but it doesn't solve the issue. The uImage works without KVM
HV in a virtual e5500 QEMU machine.

Any more progress with this? I would say that bisect might have just
been a bit unstable and maybe by chance some things did not crash so
it's pointing to the wrong patch.

Upstream merge of powerpc-5.13-1 was good and powerpc-5.13-2 was bad?

Between that looks like some KVM MMU rework. You could try the patch
before this one b1c5356e873c ("KVM: PPC: Convert to the gfn-based MMU
notifier callbacks"). That won't revert cleanly so just try run the
tree at that point. If it works, test the patch and see if it fails.

Thanks,
Nick

Christian Zigotzky

2021-05-17 09:42:45 UTC

Post by Christian Zigotzky
Hi All,
I bisected today [1] and the bisecting itself was OK but the
reverting of the bad commit doesn't solve the issue. Do you have an
idea which commit could be resposible for this issue? Maybe the
bisecting wasn't successful. I will look in the kernel git log. Maybe
there is a commit that affected KVM HV on FSL P50x0 machines.

I tried it but it doesn't solve the issue. The uImage works without KVM
HV in a virtual e5500 QEMU machine.

Any more progress with this? I would say that bisect might have just
been a bit unstable and maybe by chance some things did not crash so
it's pointing to the wrong patch.
Upstream merge of powerpc-5.13-1 was good and powerpc-5.13-2 was bad?
Between that looks like some KVM MMU rework. You could try the patch
before this one b1c5356e873c ("KVM: PPC: Convert to the gfn-based MMU
notifier callbacks"). That won't revert cleanly so just try run the
tree at that point. If it works, test the patch and see if it fails.
Thanks,
Nick

Hi Nick,

Thanks a lot for your answer. Yes, there is a little bit of progress.
The RC2 of kernel 5.13 successfully boots with -smp 3 in a virtual e5500
QEMU machine.
-smp 4 doesn't work anymore since the PowerPC updates 5.13-2. I used
-smp 4 before 5.13 because my FSL P5040 machine has 4 cores.

Could you please post a patch for reverting the commit before
b1c5356e873c ("KVM: PPC: Convert to the gfn-based MMU notifier callbacks")?

Thanks in advance,

Christian

Christian Zigotzky

2021-05-18 06:33:26 UTC

Post by Christian Zigotzky
Hi All,
I bisected today [1] and the bisecting itself was OK but the
reverting of the bad commit doesn't solve the issue. Do you have an
idea which commit could be resposible for this issue? Maybe the
bisecting wasn't successful. I will look in the kernel git log. Maybe
there is a commit that affected KVM HV on FSL P50x0 machines.

I tried it but it doesn't solve the issue. The uImage works without KVM
HV in a virtual e5500 QEMU machine.

Any more progress with this? I would say that bisect might have just
been a bit unstable and maybe by chance some things did not crash so
it's pointing to the wrong patch.
Upstream merge of powerpc-5.13-1 was good and powerpc-5.13-2 was bad?
Between that looks like some KVM MMU rework. You could try the patch
before this one b1c5356e873c ("KVM: PPC: Convert to the gfn-based MMU
notifier callbacks"). That won't revert cleanly so just try run the
tree at that point. If it works, test the patch and see if it fails.
Thanks,
Nick

Hi Nick,
Thanks a lot for your answer. Yes, there is a little bit of progress. The RC2 of kernel 5.13 successfully boots with -smp 3 in a virtual e5500 QEMU machine.
-smp 4 doesn't work anymore since the PowerPC updates 5.13-2. I used -smp 4 before 5.13 because my FSL P5040 machine has 4 cores.
Could you please post a patch for reverting the commit before b1c5356e873c ("KVM: PPC: Convert to the gfn-based MMU notifier callbacks")?
Thanks in advance,
Christian

For me it is ok to work with -smp 1, 2, and 3 but I am curious why -smp 4 doesn’t work.

-Christian

Nicholas Piggin

2021-05-19 07:57:08 UTC

Post by Christian Zigotzky
Hi All,
I bisected today [1] and the bisecting itself was OK but the
reverting of the bad commit doesn't solve the issue. Do you have an
idea which commit could be resposible for this issue? Maybe the
bisecting wasn't successful. I will look in the kernel git log. Maybe
there is a commit that affected KVM HV on FSL P50x0 machines.

I tried it but it doesn't solve the issue. The uImage works without KVM
HV in a virtual e5500 QEMU machine.

Any more progress with this? I would say that bisect might have just
been a bit unstable and maybe by chance some things did not crash so
it's pointing to the wrong patch.
Upstream merge of powerpc-5.13-1 was good and powerpc-5.13-2 was bad?
Between that looks like some KVM MMU rework. You could try the patch
before this one b1c5356e873c ("KVM: PPC: Convert to the gfn-based MMU
notifier callbacks"). That won't revert cleanly so just try run the
tree at that point. If it works, test the patch and see if it fails.
Thanks,
Nick

Hi Nick,
Thanks a lot for your answer. Yes, there is a little bit of progress.
The RC2 of kernel 5.13 successfully boots with -smp 3 in a virtual e5500
QEMU machine.
-smp 4 doesn't work anymore since the PowerPC updates 5.13-2. I used
-smp 4 before 5.13 because my FSL P5040 machine has 4 cores.
Could you please post a patch for reverting the commit before
b1c5356e873c ("KVM: PPC: Convert to the gfn-based MMU notifier callbacks")?

You could `git checkout b1c5356e873c~1`

Thanks,
Nick

Christian Zigotzky

2021-05-19 11:52:16 UTC

Post by Christian Zigotzky
I tried it but it doesn't solve the issue. The uImage works without KVM
HV in a virtual e5500 QEMU machine.

Any more progress with this? I would say that bisect might have just
been a bit unstable and maybe by chance some things did not crash so
it's pointing to the wrong patch.
Upstream merge of powerpc-5.13-1 was good and powerpc-5.13-2 was bad?
Between that looks like some KVM MMU rework. You could try the patch
before this one b1c5356e873c ("KVM: PPC: Convert to the gfn-based MMU
notifier callbacks"). That won't revert cleanly so just try run the
tree at that point. If it works, test the patch and see if it fails.
Thanks,
Nick

Hi Nick,
Thanks a lot for your answer. Yes, there is a little bit of progress.
The RC2 of kernel 5.13 successfully boots with -smp 3 in a virtual e5500
QEMU machine.
-smp 4 doesn't work anymore since the PowerPC updates 5.13-2. I used
-smp 4 before 5.13 because my FSL P5040 machine has 4 cores.
Could you please post a patch for reverting the commit before
b1c5356e873c ("KVM: PPC: Convert to the gfn-based MMU notifier callbacks")?

You could `git checkout b1c5356e873c~1`
Thanks,
Nick

Hi Nick,

Thanks for your answer. I checked out the commit b1c5356e873c~1 (HEAD is
now at d923ff258423 KVM: MIPS/MMU: Convert to the gfn-based MMU notifier
callbacks).
The kernel boots with '-smp 4' without any problems.
After that I patched with the probable first bad commit (KVM: PPC:
Convert to the gfn-based MMU notifier callbacks). The kernel also boots
with this patch. That means, this isn't the first bad commit.
Further information:
https://forum.hyperion-entertainment.com/viewtopic.php?p=53267#p53267

Cheers,
Christian

Nicholas Piggin

2021-05-19 23:07:51 UTC

Post by Christian Zigotzky
I tried it but it doesn't solve the issue. The uImage works without KVM
HV in a virtual e5500 QEMU machine.

Any more progress with this? I would say that bisect might have just
been a bit unstable and maybe by chance some things did not crash so
it's pointing to the wrong patch.
Upstream merge of powerpc-5.13-1 was good and powerpc-5.13-2 was bad?
Between that looks like some KVM MMU rework. You could try the patch
before this one b1c5356e873c ("KVM: PPC: Convert to the gfn-based MMU
notifier callbacks"). That won't revert cleanly so just try run the
tree at that point. If it works, test the patch and see if it fails.
Thanks,
Nick

Hi Nick,
Thanks a lot for your answer. Yes, there is a little bit of progress.
The RC2 of kernel 5.13 successfully boots with -smp 3 in a virtual e5500
QEMU machine.
-smp 4 doesn't work anymore since the PowerPC updates 5.13-2. I used
-smp 4 before 5.13 because my FSL P5040 machine has 4 cores.
Could you please post a patch for reverting the commit before
b1c5356e873c ("KVM: PPC: Convert to the gfn-based MMU notifier callbacks")?

You could `git checkout b1c5356e873c~1`
Thanks,
Nick

Hi Nick,
Thanks for your answer. I checked out the commit b1c5356e873c~1 (HEAD is
now at d923ff258423 KVM: MIPS/MMU: Convert to the gfn-based MMU notifier
callbacks).
The kernel boots with '-smp 4' without any problems.
Convert to the gfn-based MMU notifier callbacks). The kernel also boots
with this patch. That means, this isn't the first bad commit.
https://forum.hyperion-entertainment.com/viewtopic.php?p=53267#p53267

Hmm, okay that probably rules out those notifier changes then.

Can you remind me were you able to rule these out as suspects?

8f6cc75a97d1 powerpc: move norestart trap flag to bit 0
8dc7f0229b78 powerpc: remove partial register save logic
c45ba4f44f6b powerpc: clean up do_page_fault
d738ee8d56de powerpc/64e/interrupt: handle bad_page_fault in C
ceff77efa4f8 powerpc/64e/interrupt: Use new interrupt context tracking scheme
097157e16cf8 powerpc/64e/interrupt: reconcile irq soft-mask state in C
3db8aa10de9a powerpc/64e/interrupt: NMI save irq soft-mask state in C
0c2472de23ae powerpc/64e/interrupt: use new interrupt return
dc6231821a14 powerpc/interrupt: update common interrupt code for
4228b2c3d20e powerpc/64e/interrupt: always save nvgprs on interrupt
5a5a893c4ad8 powerpc/syscall: switch user_exit_irqoff and trace_hardirqs_off order

Thanks,
Nick

Christian Zigotzky

2021-05-30 06:49:09 UTC

Post by Christian Zigotzky
I tried it but it doesn't solve the issue. The uImage works without KVM
HV in a virtual e5500 QEMU machine.

Any more progress with this? I would say that bisect might have just
been a bit unstable and maybe by chance some things did not crash so
it's pointing to the wrong patch.
Upstream merge of powerpc-5.13-1 was good and powerpc-5.13-2 was bad?
Between that looks like some KVM MMU rework. You could try the patch
before this one b1c5356e873c ("KVM: PPC: Convert to the gfn-based MMU
notifier callbacks"). That won't revert cleanly so just try run the
tree at that point. If it works, test the patch and see if it fails.
Thanks,
Nick

Hi Nick,
Thanks a lot for your answer. Yes, there is a little bit of progress.
The RC2 of kernel 5.13 successfully boots with -smp 3 in a virtual e5500
QEMU machine.
-smp 4 doesn't work anymore since the PowerPC updates 5.13-2. I used
-smp 4 before 5.13 because my FSL P5040 machine has 4 cores.
Could you please post a patch for reverting the commit before
b1c5356e873c ("KVM: PPC: Convert to the gfn-based MMU notifier callbacks")?

You could `git checkout b1c5356e873c~1`
Thanks,
Nick

Hi Nick,
Thanks for your answer. I checked out the commit b1c5356e873c~1 (HEAD is
now at d923ff258423 KVM: MIPS/MMU: Convert to the gfn-based MMU notifier
callbacks).
The kernel boots with '-smp 4' without any problems.
Convert to the gfn-based MMU notifier callbacks). The kernel also boots
with this patch. That means, this isn't the first bad commit.
https://forum.hyperion-entertainment.com/viewtopic.php?p=53267#p53267

Hmm, okay that probably rules out those notifier changes then.
Can you remind me were you able to rule these out as suspects?
8f6cc75a97d1 powerpc: move norestart trap flag to bit 0
8dc7f0229b78 powerpc: remove partial register save logic
c45ba4f44f6b powerpc: clean up do_page_fault
d738ee8d56de powerpc/64e/interrupt: handle bad_page_fault in C
ceff77efa4f8 powerpc/64e/interrupt: Use new interrupt context tracking scheme
097157e16cf8 powerpc/64e/interrupt: reconcile irq soft-mask state in C
3db8aa10de9a powerpc/64e/interrupt: NMI save irq soft-mask state in C
0c2472de23ae powerpc/64e/interrupt: use new interrupt return
dc6231821a14 powerpc/interrupt: update common interrupt code for
4228b2c3d20e powerpc/64e/interrupt: always save nvgprs on interrupt
5a5a893c4ad8 powerpc/syscall: switch user_exit_irqoff and trace_hardirqs_off order
Thanks,
Nick

Hi Nick,

Thanks for your answer. Smp 4 still doesn't work on quad core e5500
CPUs. I use -smp 3 currently. Shall I checkout the commits above (in
your last answer) and test them? Do you prefer a commit for testing?

Thanks,
Christian

Christian Zigotzky

2021-06-02 11:26:35 UTC

Post by Nicholas Piggin
Hmm, okay that probably rules out those notifier changes then.
Can you remind me were you able to rule these out as suspects?
8f6cc75a97d1 powerpc: move norestart trap flag to bit 0
8dc7f0229b78 powerpc: remove partial register save logic
c45ba4f44f6b powerpc: clean up do_page_fault
d738ee8d56de powerpc/64e/interrupt: handle bad_page_fault in C
ceff77efa4f8 powerpc/64e/interrupt: Use new interrupt context tracking scheme
097157e16cf8 powerpc/64e/interrupt: reconcile irq soft-mask state in C
3db8aa10de9a powerpc/64e/interrupt: NMI save irq soft-mask state in C
0c2472de23ae powerpc/64e/interrupt: use new interrupt return
dc6231821a14 powerpc/interrupt: update common interrupt code for
4228b2c3d20e powerpc/64e/interrupt: always save nvgprs on interrupt
5a5a893c4ad8 powerpc/syscall: switch user_exit_irqoff and trace_hardirqs_off order
Thanks,
Nick

Hi Nick,

I tested these commits above today and all works with -smp 4. [1]

Smp 4 still doesn't work with the RC4 of kernel 5.13 on quad core e5500
CPUs with KVM HV. I use -smp 3 currently.

What shall I test next?

Thanks,
Christian

[1] https://forum.hyperion-entertainment.com/viewtopic.php?p=53367#p53367

Christian Zigotzky

2021-06-07 07:21:58 UTC