watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ksoftirqd/3:33]
Modules linked in: cfg80211 rfkill vsock_loopback vmw_vsock_virtio_transport_common vmw_vsock_vmci_transport vsock snd_seq_midi snd_seq_midi_event snd_ens1371 snd_ac97_codec intel_rapl_msr intel_rapl_common ac97_bus intel_pmc_core snd_rawmidi snd_seq snd_seq_device snd_pcm kvm_intel snd_timer snd kvm irqbypass crct10dif_pclmul crc32_pclmul soundcore ghash_clmulni_intel rapl pcspkr vmw_balloon joydev i2c_piix4 vmw_vmci xfs libcrc32c sd_mod t10_pi sr_mod cdrom sg ata_generic vmwgfx drm_ttm_helper ttm drm_kms_helper crc32c_intel syscopyarea sysfillrect sysimgblt fb_sys_fops drm ata_piix mptspi scsi_transport_spi serio_raw mptscsih e1000 libata mptbase dm_mod fuse
CPU: 3 PID: 33 Comm: ksoftirqd/3 Kdump: loaded Tainted: G        W        ---------r-  - 4.18.0-477.13.1.el8_8.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
RIP: 0010:e1000_xmit_frame+0xdb7/0x1070 [e1000]
Code: 84 c0 74 14 49 8b 83 c0 03 00 00 f6 80 d0 00 00 00 03 0f 84 f9 fb ff ff 41 0f b7 45 2a 41 8b 55 18 49 03 83 b8 0e 00 00 89 10 <e9> ea fb ff ff 31 d2 31 f6 b9 20 00 48 00 48 89 ef 4c 89 1c 24 e8
RSP: 0018:ffffb7e60118b668 EFLAGS: 00000286 ORIG_RAX: ffffffffffffff13
RAX: ffffb7e601743818 RBX: 00000000ab1da000 RCX: 00000000000000f3
RDX: 00000000000000f7 RSI: ffff893388bb2c40 RDI: ffff8933840ae000
RBP: ffff8932b8b5ce00 R08: 00000000000000f7 R09: 0000000000000000
R10: 0000000000000000 R11: ffff8933840ae000 R12: 00000000000000f6
R13: ffff893388bb2c40 R14: 0000000000002648 R15: 0000000000000042
FS:  0000000000000000(0000) GS:ffff8933f8ec0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000055a4212d1ec8 CR3: 0000000031410001 CR4: 00000000003706e0
Call Trace:
 ? dev_queue_xmit_nit+0x23a/0x250
 dev_hard_start_xmit+0xd7/0x240
 sch_direct_xmit+0x9f/0x370
 __dev_queue_xmit+0x985/0xba0
 ip_finish_output2+0x192/0x430
 ? __ip_finish_output+0xdf/0x1d0
 ip_output+0x70/0xf0
 __ip_queue_xmit+0x16d/0x410
 ? dst_alloc+0x5f/0x87
 __tcp_transmit_skb+0xa26/0xb50
 ? __alloc_skb+0x8c/0x1c0
 tcp_rcv_established+0x22e/0x5f0
 tcp_v4_do_rcv+0x137/0x200
 tcp_v4_rcv+0xcde/0xd80
 ip_protocol_deliver_rcu+0x2c/0x1d0
 ip_local_deliver_finish+0x4d/0x60
 ip_local_deliver+0x6b/0xf0
 ? tcp_v4_early_demux+0xfb/0x160
 ? ip_rcv_finish+0x171/0x430
 ip_rcv+0x27a/0x3b5
 ? packet_rcv+0x43/0x4d0
 __netif_receive_skb_core+0xc08/0xd10
 ? inet_gro_receive+0x223/0x2d0
 netif_receive_skb_internal+0x3d/0xb0
 napi_gro_receive+0x118/0x180
 e1000_clean_rx_irq+0x18e/0x470 [e1000]
 e1000_clean+0x287/0x5d0 [e1000]
 __napi_poll+0x2d/0x130
 net_rx_action+0x252/0x320
 __do_softirq+0xdc/0x2cf
 ? sort_range+0x20/0x20
 run_ksoftirqd+0x2a/0x40
 smpboot_thread_fn+0xb5/0x150
 kthread+0x134/0x150
 ? set_kthread_struct+0x50/0x50
 ret_from_fork+0x1f/0x40

watchdog: BUG: soft lockup - CPU#2 stuck for 953s! [http-nio-8080-P:2159]
rcu: INFO: rcu_sched detected stalls on CPUs/tasks:
rcu: 	3-...!: (1 ticks this GP) idle=a5e/1/0x4000000000000002 softirq=7840486/7840486 fqs=101 
Modules linked in: cfg80211
	(detected by 1, t=1046618 jiffies, g=16535557, q=91)
 rfkill vsock_loopback vmw_vsock_virtio_transport_common
Sending NMI from CPU 1 to CPUs 3:
 vmw_vsock_vmci_transport vsock snd_seq_midi snd_seq_midi_event snd_ens1371 snd_ac97_codec intel_rapl_msr intel_rapl_common ac97_bus intel_pmc_core snd_rawmidi snd_seq snd_seq_device snd_pcm kvm_intel kvm snd_timer snd irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel rapl soundcore vmw_balloon pcspkr joydev i2c_piix4 vmw_vmci xfs libcrc32c sd_mod t10_pi sr_mod cdrom sg ata_generic vmwgfx drm_ttm_helper ttm drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm ata_piix libata e1000 crc32c_intel serio_raw mptspi scsi_transport_spi mptscsih mptbase dm_mod fuse
CPU: 2 PID: 2159 Comm: http-nio-8080-P Kdump: loaded Tainted: G        W        ---------r-  - 4.18.0-477.13.1.el8_8.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
RIP: 0033:0x7f92957f5038
Code: 89 97 08 01 00 00 41 0f 0d 8a 00 01 00 00 49 c7 45 00 01 00 00 00 41 c7 45 08 80 a7 15 00 45 89 65 0c 45 89 65 10 4d 89 65 20 <4d> 89 65 28 4d 89 65 30 45 89 45 14 45 89 45 18 45 89 5d 1c 41 89
RSP: 002b:00007f921283f810 EFLAGS: 00010287 ORIG_RAX: ffffffffffffff13
RAX: 00007f9233159630 RBX: 0000000000000000 RCX: 000000008c9d1570
RDX: 0000000000000151 RSI: 000000008c9d1558 RDI: 0000000000000000
RBP: 000000008c9d1570 R08: 0000000000000040 R09: 0000000000000000
R10: 00000000a6761010 R11: 000000008f800000 R12: 0000000000000000
R13: 00000000a6760fd8 R14: 0000000000000000 R15: 00007f92a62eceb0
FS:  00007f9212840700 GS:  0000000000000000
NMI backtrace for cpu 3
CPU: 3 PID: 220562 Comm: kworker/u256:0 Kdump: loaded Tainted: G        W        ---------r-  - 4.18.0-477.13.1.el8_8.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
Workqueue: writeback wb_workfn (flush-253:0)
RIP: 0010:smp_call_function_single+0xd2/0xf0
Code: 34 25 28 00 00 00 75 3a c9 e9 aa 64 c5 00 48 89 74 24 10 48 89 e6 48 89 54 24 18 e8 a8 fe ff ff 8b 14 24 83 e2 01 74 0a f3 90 <8b> 14 24 83 e2 01 75 f6 eb c4 8b 05 36 0b 50 02 85 c0 75 84 0f 0b
RSP: 0018:ffffb5d4825af7a0 EFLAGS: 00000202
RAX: 0000000000000000 RBX: ffff9aebc8527058 RCX: 0000000000000830
RDX: 0000000000000001 RSI: 00000000000000fb RDI: 0000000000000830
RBP: ffffb5d4825af7f8 R08: 0000000000000000 R09: 0000000000000867
R10: 0000000000000001 R11: 0000000000000228 R12: ffff9aebfbd81000
R13: ffffb5d4825af944 R14: 0000000000000000 R15: 0000000000000000
FS:  0000000000000000(0000) GS:ffff9aecf5ec0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f0668242000 CR3: 0000000017010004 CR4: 00000000003706e0
Call Trace:
 ? flush_tlb_func_common.constprop.9+0x220/0x220
 ? flush_tlb_func_common.constprop.9+0x220/0x220
 flush_tlb_mm_range+0xde/0x110
 ptep_clear_flush+0x58/0x70
 page_mkclean_one+0x130/0x240
 rmap_walk_file+0xe6/0x280
 page_mkclean+0xa8/0xc0
 ? __page_set_anon_rmap+0x70/0x70
 ? pmdp_collapse_flush+0x10/0x10
 clear_page_dirty_for_io+0xa4/0x1a0
 write_cache_pages+0x14d/0x420
 ? iomap_invalidatepage+0xe0/0xe0
 iomap_writepages+0x1c/0x40
 xfs_vm_writepages+0x7e/0xb0 [xfs]
 do_writepages+0xc6/0x1d0
 __writeback_single_inode+0x39/0x300
 writeback_sb_inodes+0x1ea/0x450
 __writeback_inodes_wb+0x5f/0xd0
 wb_writeback+0x24c/0x2e0
 ? get_nr_inodes+0x35/0x60
 wb_workfn+0x380/0x4d0
 ? __smp_call_single_queue+0x23/0x40
 ? ttwu_queue_wakelist+0xf2/0x110
 process_one_work+0x1a7/0x360
 ? create_worker+0x1a0/0x1a0
 worker_thread+0x30/0x390
 ? create_worker+0x1a0/0x1a0
 kthread+0x134/0x150
 ? set_kthread_struct+0x50/0x50
 ret_from_fork+0x1f/0x40

watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ksoftirqd/2:27]
Modules linked in: cfg80211 rfkill vsock_loopback vmw_vsock_virtio_transport_common vmw_vsock_vmci_transport vsock snd_seq_midi snd_seq_midi_event snd_ens1371 snd_ac97_codec ac97_bus intel_rapl_msr snd_rawmidi intel_rapl_common snd_seq intel_pmc_core snd_seq_device snd_pcm snd_timer kvm_intel snd kvm soundcore irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel rapl pcspkr vmw_balloon joydev i2c_piix4 vmw_vmci xfs libcrc32c sr_mod cdrom sd_mod t10_pi sg ata_generic vmwgfx drm_ttm_helper ttm drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops crc32c_intel e1000 drm ata_piix mptspi serio_raw libata scsi_transport_spi mptscsih mptbase dm_mod fuse
CPU: 2 PID: 27 Comm: ksoftirqd/2 Kdump: loaded Tainted: G                 ---------r-  - 4.18.0-477.13.1.el8_8.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
RIP: 0010:e1000_xmit_frame+0xdb7/0x1070 [e1000]
Code: 84 c0 74 14 49 8b 83 c0 03 00 00 f6 80 d0 00 00 00 03 0f 84 f9 fb ff ff 41 0f b7 45 2a 41 8b 55 18 49 03 83 b8 0e 00 00 89 10 <e9> ea fb ff ff 31 d2 31 f6 b9 20 00 48 00 48 89 ef 4c 89 1c 24 e8
RSP: 0018:ffffa390011579e8 EFLAGS: 00000286 ORIG_RAX: ffffffffffffff13
RAX: ffffa390017a3818 RBX: 00000000ab188800 RCX: 0000000000000041
RDX: 000000000000003b RSI: ffff962d08e489c0 RDI: ffff962d03eb6000
RBP: ffff962dabee3400 R08: 000000000000003b R09: 0000000000000000
R10: 0000000000000000 R11: ffff962d03eb6000 R12: 000000000000003a
R13: ffff962d08e489c0 R14: 00000000000008e8 R15: 000000000000004a
FS:  0000000000000000(0000) GS:ffff962e35e80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00005630f5ca9cd8 CR3: 00000001e5c10004 CR4: 00000000003706e0
Call Trace:
 ? dev_queue_xmit_nit+0x23a/0x250
 dev_hard_start_xmit+0xd7/0x240
 sch_direct_xmit+0x9f/0x370
 __dev_queue_xmit+0x985/0xba0
 ? tcp_make_synack+0x395/0x4e0
 ip_finish_output2+0x26d/0x430
 ? __ip_finish_output+0xdf/0x1d0
 ? inet_csk_reqsk_queue_drop_and_put+0x90/0x90
 ip_output+0x70/0xf0
 ? ip_build_and_send_pkt+0x4a/0x1f0
 tcp_v4_send_synack+0x70/0xe0
 tcp_rtx_synack+0x64/0xb0
 ? internal_add_timer+0x42/0x70
 reqsk_timer_handler+0x1f9/0x2a0
 ? inet_csk_reqsk_queue_drop_and_put+0x90/0x90
 call_timer_fn+0x2e/0x130
 run_timer_softirq+0x1d8/0x410
 ? __switch_to_asm+0x43/0x80
 ? __switch_to+0x10c/0x450
 ? sched_clock+0x5/0x10
 __do_softirq+0xdc/0x2cf
 ? sort_range+0x20/0x20
 run_ksoftirqd+0x2a/0x40
 smpboot_thread_fn+0xb5/0x150
 kthread+0x134/0x150
 ? set_kthread_struct+0x50/0x50
 ret_from_fork+0x1f/0x40