kernel BUG at mm/huge_memory.c:1798!

classic Classic list List threaded Threaded
41 messages Options
123
Reply | Threaded
Open this post in threaded view
|

kernel BUG at mm/huge_memory.c:1798!

Zhouping Liu
Hello all,

I found the below kernel bug using latest mainline(637704cbc95),
my hardware has 2 numa nodes, and it's easy to reproduce the issue
using LTP test case: "# ./mmap10 -a -s -c 200":

[root@localhost linux]# cat .config | grep NUMA
CONFIG_ARCH_SUPPORTS_NUMA_BALANCING=y
CONFIG_ARCH_WANTS_PROT_NUMA_PROT_NONE=y
CONFIG_ARCH_USES_NUMA_PROT_NONE=y
CONFIG_NUMA_BALANCING_DEFAULT_ENABLED=y
CONFIG_NUMA_BALANCING=y
# CONFIG_X86_NUMACHIP is not set
CONFIG_NUMA=y
CONFIG_AMD_NUMA=y
CONFIG_X86_64_ACPI_NUMA=y
# CONFIG_NUMA_EMU is not set
CONFIG_USE_PERCPU_NUMA_NODE_ID=y
CONFIG_ACPI_NUMA=y

-----------------------------------------------------------
[  588.143072] mapcount 0 page_mapcount 3
[  588.147471] ------------[ cut here ]------------
[  588.152856] kernel BUG at mm/huge_memory.c:1798!
[  588.158125] invalid opcode: 0000 [#1] SMP
[  588.162882] Modules linked in: ip6table_filter ip6_tables ebtable_nat ebtables bnep bluetooth rfkill iptable_mangle ipt_REJECT nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack iptable_filter ip_tables be2iscsi iscsi_boot_sysfs bnx2i cnic uio cxgb4i cxgb4 cxgb3i cxgb3 mdio libcxgbi ib_iser rdma_cm ib_addr iw_cm ib_cm ib_sa ib_mad ib_core iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi vfat fat dm_mirror dm_region_hash dm_log dm_mod cdc_ether iTCO_wdt i7core_edac coretemp usbnet iTCO_vendor_support mii crc32c_intel edac_core lpc_ich shpchp ioatdma mfd_core i2c_i801 pcspkr serio_raw bnx2 microcode dca vhost_net tun macvtap macvlan kvm_intel kvm uinput mgag200 sr_mod cdrom i2c_algo_bit sd_mod drm_kms_helper crc_t10dif ata_generic pata_acpi ttm ata_piix drm libata i2c_core megaraid_sas

[  588.246517] CPU 1
[  588.248636] Pid: 23217, comm: mmap10 Not tainted 3.8.0-rc1mainline+ #17 IBM IBM System x3400 M3 Server -[7379I08]-/69Y4356    
[  588.262171] RIP: 0010:[<ffffffff8118fac7>]  [<ffffffff8118fac7>] __split_huge_page+0x677/0x6d0
[  588.272067] RSP: 0000:ffff88017a03fc08  EFLAGS: 00010293
[  588.278235] RAX: 0000000000000003 RBX: ffff88027a6c22e0 RCX: 00000000000034d2
[  588.286394] RDX: 000000000000748b RSI: 0000000000000046 RDI: 0000000000000246
[  588.294216] RBP: ffff88017a03fcb8 R08: ffffffff819d2440 R09: 000000000000054a
[  588.302441] R10: 0000000000aaaaaa R11: 00000000ffffffff R12: 0000000000000000
[  588.310495] R13: 00007f4f11a00000 R14: ffff880179e96e00 R15: ffffea0005c08000
[  588.318640] FS:  00007f4f11f4a740(0000) GS:ffff88017bc20000(0000) knlGS:0000000000000000
[  588.327894] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[  588.334569] CR2: 00000037e9ebb404 CR3: 000000017a436000 CR4: 00000000000007e0
[  588.342718] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[  588.350861] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[  588.359134] Process mmap10 (pid: 23217, threadinfo ffff88017a03e000, task ffff880172dd32e0)
[  588.368667] Stack:
[  588.370960]  ffff88017a540ec8 ffff88017a03fc20 ffffffff816017b5 ffff88017a03fc88
[  588.379566]  ffffffff812fa014 0000000000000000 ffff880279ebd5c0 00000000f4f11a4c
[  588.388150]  00000007f4f11f49 00000007f4f11a00 ffff88017a540ef0 ffff88017a540ee8
[  588.396711] Call Trace:
[  588.455106]  [<ffffffff816017b5>] ? rwsem_down_read_failed+0x15/0x17
[  588.518106]  [<ffffffff812fa014>] ? call_rwsem_down_read_failed+0x14/0x30
[  588.580897]  [<ffffffff815ffc04>] ? down_read+0x24/0x2b
[  588.642630]  [<ffffffff8118fb88>] split_huge_page+0x68/0xb0
[  588.703814]  [<ffffffff81190ed4>] __split_huge_page_pmd+0x134/0x330
[  588.766064]  [<ffffffff8104b997>] ? pte_alloc_one+0x37/0x50
[  588.826460]  [<ffffffff81191121>] split_huge_page_pmd_mm+0x51/0x60
[  588.887746]  [<ffffffff8119116b>] split_huge_page_address+0x3b/0x50
[  588.948673]  [<ffffffff8119121c>] __vma_adjust_trans_huge+0x9c/0xf0
[  589.008660]  [<ffffffff811650f4>] vma_adjust+0x684/0x750
[  589.066328]  [<ffffffff811653ba>] __split_vma.isra.28+0x1fa/0x220
[  589.123497]  [<ffffffff810135d1>] ? __switch_to+0x181/0x4a0
[  589.180704]  [<ffffffff811661a9>] do_munmap+0xf9/0x420
[  589.237461]  [<ffffffff8160026c>] ? __schedule+0x3cc/0x7b0
[  589.294520]  [<ffffffff8116651e>] vm_munmap+0x4e/0x70
[  589.350784]  [<ffffffff8116741b>] sys_munmap+0x2b/0x40
[  589.406971]  [<ffffffff8160a159>] system_call_fastpath+0x16/0x1b
[  589.464792] Code: 49 8b 07 a9 00 00 00 01 75 f4 e9 2d fb ff ff 41 8b 4f 18 8b 75 8c 48 c7 c7 f8 27 81 81 31 c0 83 c1 01 e8 df 63 46 00 0f 0b 0f 0b <0f> 0b 41 8b 57 18 8b 75 8c 48 c7 c7 d8 27 81 81 31 c0 83 c2 01
[  589.595165] RIP  [<ffffffff8118fac7>] __split_huge_page+0x677/0x6d0
[  589.656000]  RSP <ffff88017a03fc08>
[  589.713937] ---[ end trace bff29bee67936f30 ]---


--
Thanks,
Zhouping
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [hidden email]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/
Reply | Threaded
Open this post in threaded view
|

Re: kernel BUG at mm/huge_memory.c:1798!

Hillf Danton
On Tue, Dec 25, 2012 at 12:38 PM, Zhouping Liu <[hidden email]> wrote:
> Hello all,
>
> I found the below kernel bug using latest mainline(637704cbc95),
> my hardware has 2 numa nodes, and it's easy to reproduce the issue
> using LTP test case: "# ./mmap10 -a -s -c 200":

Can you test with 5a505085f0 and 4fc3f1d66b1 reverted?

Hillf
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [hidden email]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/
Reply | Threaded
Open this post in threaded view
|

Re: kernel BUG at mm/huge_memory.c:1798!

Zhouping Liu
On 12/25/2012 08:05 PM, Hillf Danton wrote:
> On Tue, Dec 25, 2012 at 12:38 PM, Zhouping Liu <[hidden email]> wrote:
>> Hello all,
>>
>> I found the below kernel bug using latest mainline(637704cbc95),
>> my hardware has 2 numa nodes, and it's easy to reproduce the issue
>> using LTP test case: "# ./mmap10 -a -s -c 200":
> Can you test with 5a505085f0 and 4fc3f1d66b1 reverted?

Hello Hillf, I tested it with 5a505085f0 and 4fc3f1d66b1 reverted, you
are right, the issue is gone.

Thanks,
Zhouping
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [hidden email]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/
Reply | Threaded
Open this post in threaded view
|

BUG: unable to handle kernel NULL pointer dereference at 0000000000000500

Zhouping Liu
In reply to this post by Zhouping Liu
Hello everyone,

The latest mainline(637704cbc95c) would trigger the following error when the system was under
some pressure condition(in my testing, I used oom01 case inside LTP test suite to trigger the issue):

[ 5462.920151] BUG: unable to handle kernel NULL pointer dereference at 0000000000000500
[ 5462.927991] IP: [<ffffffff811542d9>] wait_iff_congested+0x59/0x140
[ 5462.934176] PGD 0
[ 5462.936191] Oops: 0000 [#2] SMP
[ 5462.939428] Modules linked in: lockd sunrpc iptable_mangle ipt_REJECT nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_tabled
[ 5462.984261] CPU 13
[ 5462.986184] Pid: 117, comm: kswapd3 Tainted: G      D      3.8.0-rc1+ #1 Dell Inc. PowerEdge M905/0D413F
[ 5462.995814] RIP: 0010:[<ffffffff811542d9>]  [<ffffffff811542d9>] wait_iff_congested+0x59/0x140
[ 5463.004411] RSP: 0018:ffff88007c97fd48  EFLAGS: 00010202
[ 5463.009701] RAX: 0000000000000001 RBX: 0000000000000064 RCX: 0000000000000001
[ 5463.016818] RDX: 0000000000000064 RSI: 0000000000000000 RDI: 0000000000000000
[ 5463.023926] RBP: ffff88007c97fd98 R08: 0000000000000000 R09: ffff88022ffd9d80
[ 5463.031033] R10: 0000000000003189 R11: 0000000000000000 R12: 00000001004ee87e
[ 5463.038140] R13: 0000000000000002 R14: 0000000000000000 R15: ffff88022ffd9000
[ 5463.045258] FS:  00007f3e570de740(0000) GS:ffff88022fcc0000(0000) knlGS:0000000000000000
[ 5463.053317] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 5463.059041] CR2: 0000000000000500 CR3: 00000000018dc000 CR4: 00000000000007e0
[ 5463.066157] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 5463.073276] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 5463.080400] Process kswapd3 (pid: 117, threadinfo ffff88007c97e000, task ffff88007c981970)
[ 5463.088633] Stack:
[ 5463.090646]  ffff88007c97fd98 0000000000000000 ffff88007c981970 ffffffff81086080
[ 5463.098090]  ffff88007c97fd68 ffff88007c97fd68 ffff88022ffd9d80 0000000000000002
[ 5463.105527]  0000000000000002 0000000000000000 ffff88007c97feb8 ffffffff8114b0e3
[ 5463.112998] Call Trace:
[ 5463.115446]  [<ffffffff81086080>] ? wake_up_bit+0x40/0x40
[ 5463.120826]  [<ffffffff8114b0e3>] kswapd+0x6c3/0xa50
[ 5463.125775]  [<ffffffff8114aa20>] ? zone_reclaim+0x270/0x270
[ 5463.131415]  [<ffffffff81085680>] kthread+0xc0/0xd0
[ 5463.136278]  [<ffffffff810855c0>] ? kthread_create_on_node+0x120/0x120
[ 5463.142786]  [<ffffffff8160a0ac>] ret_from_fork+0x7c/0xb0
[ 5463.148166]  [<ffffffff810855c0>] ? kthread_create_on_node+0x120/0x120
[ 5463.154668] Code: 4e 6d 88 00 48 c7 45 b8 00 00 00 00 48 83 c0 18 48 c7 45 c8 80 60 08 81 48 89 45 d0 48 89 45 d8 8b 04 b5 a0 9a cd 81 85 c0 74 0f <48> 8b 87 00 05 00 00 a8 04 0f 85 98 00 00 00 e8 b3 c3  
[ 5463.174097] RIP  [<ffffffff811542d9>] wait_iff_congested+0x59/0x140
[ 5463.180352]  RSP <ffff88007c97fd48>
[ 5463.183824] CR2: 0000000000000500
[ 5463.203717] ---[ end trace 9ff4ff9087c13a36 ]---

I attached the config file, hope it can make some help.

Thanks,
Zhouping

config (150K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: BUG: unable to handle kernel NULL pointer dereference at 0000000000000500

Hillf Danton
On Wed, Dec 26, 2012 at 7:22 PM, Zhouping Liu <[hidden email]> wrote:

> Hello everyone,
>
> The latest mainline(637704cbc95c) would trigger the following error when the system was under
> some pressure condition(in my testing, I used oom01 case inside LTP test suite to trigger the issue):
>
> [ 5462.920151] BUG: unable to handle kernel NULL pointer dereference at 0000000000000500
> [ 5462.927991] IP: [<ffffffff811542d9>] wait_iff_congested+0x59/0x140
> [ 5462.934176] PGD 0
> [ 5462.936191] Oops: 0000 [#2] SMP
> [ 5462.939428] Modules linked in: lockd sunrpc iptable_mangle ipt_REJECT nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_tabled
> [ 5462.984261] CPU 13
> [ 5462.986184] Pid: 117, comm: kswapd3 Tainted: G      D      3.8.0-rc1+ #1 Dell Inc. PowerEdge M905/0D413F
> [ 5462.995814] RIP: 0010:[<ffffffff811542d9>]  [<ffffffff811542d9>] wait_iff_congested+0x59/0x140
> [ 5463.004411] RSP: 0018:ffff88007c97fd48  EFLAGS: 00010202
> [ 5463.009701] RAX: 0000000000000001 RBX: 0000000000000064 RCX: 0000000000000001
> [ 5463.016818] RDX: 0000000000000064 RSI: 0000000000000000 RDI: 0000000000000000
> [ 5463.023926] RBP: ffff88007c97fd98 R08: 0000000000000000 R09: ffff88022ffd9d80
> [ 5463.031033] R10: 0000000000003189 R11: 0000000000000000 R12: 00000001004ee87e
> [ 5463.038140] R13: 0000000000000002 R14: 0000000000000000 R15: ffff88022ffd9000
> [ 5463.045258] FS:  00007f3e570de740(0000) GS:ffff88022fcc0000(0000) knlGS:0000000000000000
> [ 5463.053317] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [ 5463.059041] CR2: 0000000000000500 CR3: 00000000018dc000 CR4: 00000000000007e0
> [ 5463.066157] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> [ 5463.073276] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> [ 5463.080400] Process kswapd3 (pid: 117, threadinfo ffff88007c97e000, task ffff88007c981970)
> [ 5463.088633] Stack:
> [ 5463.090646]  ffff88007c97fd98 0000000000000000 ffff88007c981970 ffffffff81086080
> [ 5463.098090]  ffff88007c97fd68 ffff88007c97fd68 ffff88022ffd9d80 0000000000000002
> [ 5463.105527]  0000000000000002 0000000000000000 ffff88007c97feb8 ffffffff8114b0e3
> [ 5463.112998] Call Trace:
> [ 5463.115446]  [<ffffffff81086080>] ? wake_up_bit+0x40/0x40
> [ 5463.120826]  [<ffffffff8114b0e3>] kswapd+0x6c3/0xa50
> [ 5463.125775]  [<ffffffff8114aa20>] ? zone_reclaim+0x270/0x270
> [ 5463.131415]  [<ffffffff81085680>] kthread+0xc0/0xd0
> [ 5463.136278]  [<ffffffff810855c0>] ? kthread_create_on_node+0x120/0x120
> [ 5463.142786]  [<ffffffff8160a0ac>] ret_from_fork+0x7c/0xb0
> [ 5463.148166]  [<ffffffff810855c0>] ? kthread_create_on_node+0x120/0x120
> [ 5463.154668] Code: 4e 6d 88 00 48 c7 45 b8 00 00 00 00 48 83 c0 18 48 c7 45 c8 80 60 08 81 48 89 45 d0 48 89 45 d8 8b 04 b5 a0 9a cd 81 85 c0 74 0f <48> 8b 87 00 05 00 00 a8 04 0f 85 98 00 00 00 e8 b3 c3
> [ 5463.174097] RIP  [<ffffffff811542d9>] wait_iff_congested+0x59/0x140
> [ 5463.180352]  RSP <ffff88007c97fd48>
> [ 5463.183824] CR2: 0000000000000500
> [ 5463.203717] ---[ end trace 9ff4ff9087c13a36 ]---
>
> I attached the config file, hope it can make some help.

Hey Zhouping,

Can you please run with the following commit reverted?

Hillf

   commit cda73a10eb3f
   mm: do not sleep in balance_pgdat if there's no i/o congestion
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [hidden email]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/
Reply | Threaded
Open this post in threaded view
|

Re: BUG: unable to handle kernel NULL pointer dereference at 0000000000000500

Zhouping Liu
On 12/26/2012 08:01 PM, Hillf Danton wrote:

> On Wed, Dec 26, 2012 at 7:22 PM, Zhouping Liu <[hidden email]> wrote:
>> Hello everyone,
>>
>> The latest mainline(637704cbc95c) would trigger the following error when the system was under
>> some pressure condition(in my testing, I used oom01 case inside LTP test suite to trigger the issue):
>>
>> [ 5462.920151] BUG: unable to handle kernel NULL pointer dereference at 0000000000000500
>> [ 5462.927991] IP: [<ffffffff811542d9>] wait_iff_congested+0x59/0x140
>> [ 5462.934176] PGD 0
>> [ 5462.936191] Oops: 0000 [#2] SMP
>> [ 5462.939428] Modules linked in: lockd sunrpc iptable_mangle ipt_REJECT nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_tabled
>> [ 5462.984261] CPU 13
>> [ 5462.986184] Pid: 117, comm: kswapd3 Tainted: G      D      3.8.0-rc1+ #1 Dell Inc. PowerEdge M905/0D413F
>> [ 5462.995814] RIP: 0010:[<ffffffff811542d9>]  [<ffffffff811542d9>] wait_iff_congested+0x59/0x140
>> [ 5463.004411] RSP: 0018:ffff88007c97fd48  EFLAGS: 00010202
>> [ 5463.009701] RAX: 0000000000000001 RBX: 0000000000000064 RCX: 0000000000000001
>> [ 5463.016818] RDX: 0000000000000064 RSI: 0000000000000000 RDI: 0000000000000000
>> [ 5463.023926] RBP: ffff88007c97fd98 R08: 0000000000000000 R09: ffff88022ffd9d80
>> [ 5463.031033] R10: 0000000000003189 R11: 0000000000000000 R12: 00000001004ee87e
>> [ 5463.038140] R13: 0000000000000002 R14: 0000000000000000 R15: ffff88022ffd9000
>> [ 5463.045258] FS:  00007f3e570de740(0000) GS:ffff88022fcc0000(0000) knlGS:0000000000000000
>> [ 5463.053317] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
>> [ 5463.059041] CR2: 0000000000000500 CR3: 00000000018dc000 CR4: 00000000000007e0
>> [ 5463.066157] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
>> [ 5463.073276] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
>> [ 5463.080400] Process kswapd3 (pid: 117, threadinfo ffff88007c97e000, task ffff88007c981970)
>> [ 5463.088633] Stack:
>> [ 5463.090646]  ffff88007c97fd98 0000000000000000 ffff88007c981970 ffffffff81086080
>> [ 5463.098090]  ffff88007c97fd68 ffff88007c97fd68 ffff88022ffd9d80 0000000000000002
>> [ 5463.105527]  0000000000000002 0000000000000000 ffff88007c97feb8 ffffffff8114b0e3
>> [ 5463.112998] Call Trace:
>> [ 5463.115446]  [<ffffffff81086080>] ? wake_up_bit+0x40/0x40
>> [ 5463.120826]  [<ffffffff8114b0e3>] kswapd+0x6c3/0xa50
>> [ 5463.125775]  [<ffffffff8114aa20>] ? zone_reclaim+0x270/0x270
>> [ 5463.131415]  [<ffffffff81085680>] kthread+0xc0/0xd0
>> [ 5463.136278]  [<ffffffff810855c0>] ? kthread_create_on_node+0x120/0x120
>> [ 5463.142786]  [<ffffffff8160a0ac>] ret_from_fork+0x7c/0xb0
>> [ 5463.148166]  [<ffffffff810855c0>] ? kthread_create_on_node+0x120/0x120
>> [ 5463.154668] Code: 4e 6d 88 00 48 c7 45 b8 00 00 00 00 48 83 c0 18 48 c7 45 c8 80 60 08 81 48 89 45 d0 48 89 45 d8 8b 04 b5 a0 9a cd 81 85 c0 74 0f <48> 8b 87 00 05 00 00 a8 04 0f 85 98 00 00 00 e8 b3 c3
>> [ 5463.174097] RIP  [<ffffffff811542d9>] wait_iff_congested+0x59/0x140
>> [ 5463.180352]  RSP <ffff88007c97fd48>
>> [ 5463.183824] CR2: 0000000000000500
>> [ 5463.203717] ---[ end trace 9ff4ff9087c13a36 ]---
>>
>> I attached the config file, hope it can make some help.
> Hey Zhouping,
>
> Can you please run with the following commit reverted?
>
> Hillf
>
>     commit cda73a10eb3f
>     mm: do not sleep in balance_pgdat if there's no i/o congestion

Hello Hillf,

Tested it with cda73a10eb3f reverted on two machines, no such issue
occurred again.

Thanks,
Zhouping
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [hidden email]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/
Reply | Threaded
Open this post in threaded view
|

Re: BUG: unable to handle kernel NULL pointer dereference at 0000000000000500

Zlatko Calusic
In reply to this post by Zhouping Liu
On 26.12.2012 12:22, Zhouping Liu wrote:

> Hello everyone,
>
> The latest mainline(637704cbc95c) would trigger the following error when the system was under
> some pressure condition(in my testing, I used oom01 case inside LTP test suite to trigger the issue):
>
> [ 5462.920151] BUG: unable to handle kernel NULL pointer dereference at 0000000000000500
> [ 5462.927991] IP: [<ffffffff811542d9>] wait_iff_congested+0x59/0x140
> [ 5462.934176] PGD 0
> [ 5462.936191] Oops: 0000 [#2] SMP
> [ 5462.939428] Modules linked in: lockd sunrpc iptable_mangle ipt_REJECT nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_tabled
> [ 5462.984261] CPU 13
> [ 5462.986184] Pid: 117, comm: kswapd3 Tainted: G      D      3.8.0-rc1+ #1 Dell Inc. PowerEdge M905/0D413F
> [ 5462.995814] RIP: 0010:[<ffffffff811542d9>]  [<ffffffff811542d9>] wait_iff_congested+0x59/0x140
> [ 5463.004411] RSP: 0018:ffff88007c97fd48  EFLAGS: 00010202
> [ 5463.009701] RAX: 0000000000000001 RBX: 0000000000000064 RCX: 0000000000000001
> [ 5463.016818] RDX: 0000000000000064 RSI: 0000000000000000 RDI: 0000000000000000
> [ 5463.023926] RBP: ffff88007c97fd98 R08: 0000000000000000 R09: ffff88022ffd9d80
> [ 5463.031033] R10: 0000000000003189 R11: 0000000000000000 R12: 00000001004ee87e
> [ 5463.038140] R13: 0000000000000002 R14: 0000000000000000 R15: ffff88022ffd9000
> [ 5463.045258] FS:  00007f3e570de740(0000) GS:ffff88022fcc0000(0000) knlGS:0000000000000000
> [ 5463.053317] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [ 5463.059041] CR2: 0000000000000500 CR3: 00000000018dc000 CR4: 00000000000007e0
> [ 5463.066157] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> [ 5463.073276] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> [ 5463.080400] Process kswapd3 (pid: 117, threadinfo ffff88007c97e000, task ffff88007c981970)
> [ 5463.088633] Stack:
> [ 5463.090646]  ffff88007c97fd98 0000000000000000 ffff88007c981970 ffffffff81086080
> [ 5463.098090]  ffff88007c97fd68 ffff88007c97fd68 ffff88022ffd9d80 0000000000000002
> [ 5463.105527]  0000000000000002 0000000000000000 ffff88007c97feb8 ffffffff8114b0e3
> [ 5463.112998] Call Trace:
> [ 5463.115446]  [<ffffffff81086080>] ? wake_up_bit+0x40/0x40
> [ 5463.120826]  [<ffffffff8114b0e3>] kswapd+0x6c3/0xa50
> [ 5463.125775]  [<ffffffff8114aa20>] ? zone_reclaim+0x270/0x270
> [ 5463.131415]  [<ffffffff81085680>] kthread+0xc0/0xd0
> [ 5463.136278]  [<ffffffff810855c0>] ? kthread_create_on_node+0x120/0x120
> [ 5463.142786]  [<ffffffff8160a0ac>] ret_from_fork+0x7c/0xb0
> [ 5463.148166]  [<ffffffff810855c0>] ? kthread_create_on_node+0x120/0x120
> [ 5463.154668] Code: 4e 6d 88 00 48 c7 45 b8 00 00 00 00 48 83 c0 18 48 c7 45 c8 80 60 08 81 48 89 45 d0 48 89 45 d8 8b 04 b5 a0 9a cd 81 85 c0 74 0f <48> 8b 87 00 05 00 00 a8 04 0f 85 98 00 00 00 e8 b3 c3
> [ 5463.174097] RIP  [<ffffffff811542d9>] wait_iff_congested+0x59/0x140
> [ 5463.180352]  RSP <ffff88007c97fd48>
> [ 5463.183824] CR2: 0000000000000500
> [ 5463.203717] ---[ end trace 9ff4ff9087c13a36 ]---
>
> I attached the config file, hope it can make some help.
>
> Thanks,
> Zhouping
>

If I'm decoding it properly, this translates to:

0xffffffff811542e9 is in wait_iff_congested
(/usr/src/linux/arch/x86/include/asm/bitops.h:321).
316 }
317
318 static __always_inline int constant_test_bit(unsigned int nr, const
volatile unsigned long *addr)
319 {
320 return ((1UL << (nr % BITS_PER_LONG)) &
321 (addr[nr / BITS_PER_LONG])) != 0;
322 }
323
324 static inline int variable_test_bit(int nr, volatile const unsigned
long *addr)
325 {

0xffffffff811542e8 is in wait_iff_congested (mm/backing-dev.c:815).
810 /*
811 * If there is no congestion, or heavy congestion is not being
812 * encountered in the current zone, yield if necessary instead
813 * of sleeping on the congestion queue
814 */
815 if (atomic_read(&nr_bdi_congested[sync]) == 0 ||
816 !zone_is_reclaim_congested(zone)) {
817 cond_resched();
818
819 /* In case we scheduled, work out time remaining */

All code
========
    0: 4e 6d                 rex.WRX insl (%dx),%es:(%rdi)
    2: 88 00                 mov    %al,(%rax)
    4: 48 c7 45 b8 00 00 00 movq   $0x0,-0x48(%rbp)
    b: 00
    c: 48 83 c0 18           add    $0x18,%rax
   10: 48 c7 45 c8 80 60 08 movq   $0xffffffff81086080,-0x38(%rbp)
   17: 81
   18: 48 89 45 d0           mov    %rax,-0x30(%rbp)
   1c: 48 89 45 d8           mov    %rax,-0x28(%rbp)
   20: 8b 04 b5 a0 9a cd 81 mov    -0x7e326560(,%rsi,4),%eax
   27: 85 c0                 test   %eax,%eax
   29: 74 0f                 je     0x3a
   2b:* 48 8b 87 00 05 00 00 mov    0x500(%rdi),%rax     <-- trapping
instruction
   32: a8 04                 test   $0x4,%al
   34: 0f 85 98 00 00 00     jne    0xd2
   3a: e8                   .byte 0xe8
   3b: b3 c3                 mov    $0xc3,%bl

I remember when I was instrumenting vmscan.c to see which of the
congestion_wait() calls was making trouble, the only place that really
called it many times (other counters being zero all the time!) was the
one that I eventually replaced with wait_iff_congested().

So, I wonder did we now uncover a subtle bug in wait_iff_congested()
that has gone unnoticed for a long time?
--
Zlatko
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [hidden email]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/
Reply | Threaded
Open this post in threaded view
|

Re: BUG: unable to handle kernel NULL pointer dereference at 0000000000000500

Hillf Danton
In reply to this post by Zhouping Liu
On Wed, Dec 26, 2012 at 9:24 PM, Zhouping Liu <[hidden email]> wrote:

> On 12/26/2012 08:01 PM, Hillf Danton wrote:
>>
>> On Wed, Dec 26, 2012 at 7:22 PM, Zhouping Liu <[hidden email]> wrote:
>>>
>>> Hello everyone,
>>>
>>> The latest mainline(637704cbc95c) would trigger the following error when
>>> the system was under
>>> some pressure condition(in my testing, I used oom01 case inside LTP test
>>> suite to trigger the issue):
>>>
>>> [ 5462.920151] BUG: unable to handle kernel NULL pointer dereference at
>>> 0000000000000500
>>> [ 5462.927991] IP: [<ffffffff811542d9>] wait_iff_congested+0x59/0x140
>>> [ 5462.934176] PGD 0
>>> [ 5462.936191] Oops: 0000 [#2] SMP
>>> [ 5462.939428] Modules linked in: lockd sunrpc iptable_mangle ipt_REJECT
>>> nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ebtable_filter
>>> ebtables ip6table_filter ip6_tables iptable_filter ip_tabled
>>> [ 5462.984261] CPU 13
>>> [ 5462.986184] Pid: 117, comm: kswapd3 Tainted: G      D      3.8.0-rc1+
>>> #1 Dell Inc. PowerEdge M905/0D413F
>>> [ 5462.995814] RIP: 0010:[<ffffffff811542d9>]  [<ffffffff811542d9>]
>>> wait_iff_congested+0x59/0x140
>>> [ 5463.004411] RSP: 0018:ffff88007c97fd48  EFLAGS: 00010202
>>> [ 5463.009701] RAX: 0000000000000001 RBX: 0000000000000064 RCX:
>>> 0000000000000001
>>> [ 5463.016818] RDX: 0000000000000064 RSI: 0000000000000000 RDI:
>>> 0000000000000000
>>> [ 5463.023926] RBP: ffff88007c97fd98 R08: 0000000000000000 R09:
>>> ffff88022ffd9d80
>>> [ 5463.031033] R10: 0000000000003189 R11: 0000000000000000 R12:
>>> 00000001004ee87e
>>> [ 5463.038140] R13: 0000000000000002 R14: 0000000000000000 R15:
>>> ffff88022ffd9000
>>> [ 5463.045258] FS:  00007f3e570de740(0000) GS:ffff88022fcc0000(0000)
>>> knlGS:0000000000000000
>>> [ 5463.053317] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
>>> [ 5463.059041] CR2: 0000000000000500 CR3: 00000000018dc000 CR4:
>>> 00000000000007e0
>>> [ 5463.066157] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
>>> 0000000000000000
>>> [ 5463.073276] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
>>> 0000000000000400
>>> [ 5463.080400] Process kswapd3 (pid: 117, threadinfo ffff88007c97e000,
>>> task ffff88007c981970)
>>> [ 5463.088633] Stack:
>>> [ 5463.090646]  ffff88007c97fd98 0000000000000000 ffff88007c981970
>>> ffffffff81086080
>>> [ 5463.098090]  ffff88007c97fd68 ffff88007c97fd68 ffff88022ffd9d80
>>> 0000000000000002
>>> [ 5463.105527]  0000000000000002 0000000000000000 ffff88007c97feb8
>>> ffffffff8114b0e3
>>> [ 5463.112998] Call Trace:
>>> [ 5463.115446]  [<ffffffff81086080>] ? wake_up_bit+0x40/0x40
>>> [ 5463.120826]  [<ffffffff8114b0e3>] kswapd+0x6c3/0xa50
>>> [ 5463.125775]  [<ffffffff8114aa20>] ? zone_reclaim+0x270/0x270
>>> [ 5463.131415]  [<ffffffff81085680>] kthread+0xc0/0xd0
>>> [ 5463.136278]  [<ffffffff810855c0>] ? kthread_create_on_node+0x120/0x120
>>> [ 5463.142786]  [<ffffffff8160a0ac>] ret_from_fork+0x7c/0xb0
>>> [ 5463.148166]  [<ffffffff810855c0>] ? kthread_create_on_node+0x120/0x120
>>> [ 5463.154668] Code: 4e 6d 88 00 48 c7 45 b8 00 00 00 00 48 83 c0 18 48
>>> c7 45 c8 80 60 08 81 48 89 45 d0 48 89 45 d8 8b 04 b5 a0 9a cd 81 85 c0 74
>>> 0f <48> 8b 87 00 05 00 00 a8 04 0f 85 98 00 00 00 e8 b3 c3
>>> [ 5463.174097] RIP  [<ffffffff811542d9>] wait_iff_congested+0x59/0x140
>>> [ 5463.180352]  RSP <ffff88007c97fd48>
>>> [ 5463.183824] CR2: 0000000000000500
>>> [ 5463.203717] ---[ end trace 9ff4ff9087c13a36 ]---
>>>
>>> I attached the config file, hope it can make some help.
>>
>> Hey Zhouping,
>>
>> Can you please run with the following commit reverted?
>>
>> Hillf
>>
>>     commit cda73a10eb3f
>>     mm: do not sleep in balance_pgdat if there's no i/o congestion
>
>
> Hello Hillf,
>
> Tested it with cda73a10eb3f reverted on two machines, no such issue occurred
> again.

It is really good news;) thank you a ton for testing.

Hillf
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [hidden email]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/
Reply | Threaded
Open this post in threaded view
|

Re: kernel BUG at mm/huge_memory.c:1798!

Alexander Beregalov
In reply to this post by Hillf Danton
On 25 December 2012 16:05, Hillf Danton <[hidden email]> wrote:
> On Tue, Dec 25, 2012 at 12:38 PM, Zhouping Liu <[hidden email]> wrote:
>> Hello all,
>>
>> I found the below kernel bug using latest mainline(637704cbc95),
>> my hardware has 2 numa nodes, and it's easy to reproduce the issue
>> using LTP test case: "# ./mmap10 -a -s -c 200":
>
> Can you test with 5a505085f0 and 4fc3f1d66b1 reverted?
>

Hello,
does it look like the same problem?

mapcount 0 page_mapcount 1
------------[ cut here ]------------
kernel BUG at mm/huge_memory.c:1798!
invalid opcode: 0000 [#1] PREEMPT SMP
Modules linked in: r8169 radeon cfbfillrect cfbimgblt cfbcopyarea
i2c_algo_bit backlight drm_kms_helper ttm drm agpgart
CPU 3
Pid: 15825, comm: firefox Not tainted 3.8.0-rc1-00004-g637704c #1
Gigabyte Technology Co., Ltd. P35-DS3/P35-DS3
RIP: 0010:[<ffffffff810e89c9>]  [<ffffffff810e89c9>] split_huge_page+0x739/0x7a0
RSP: 0018:ffff880193b43b78  EFLAGS: 00010297
RAX: 0000000000000001 RBX: ffffea0002fd0000 RCX: ffffffff8175e078
RDX: 000000000000003e RSI: ffffea0002fd0000 RDI: 0000000000000246
RBP: ffff880193b43c48 R08: 000000000000ffff R09: 0000000000000000
R10: 00000000000002d5 R11: 0000000000000000 R12: 0000000000000000
R13: ffff880173533464 R14: 00007f0973000000 R15: ffffea0002fd0000
FS:  00007f09b8db6740(0000) GS:ffff88019fd80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007ff210e78008 CR3: 0000000195379000 CR4: 00000000000007e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process firefox (pid: 15825, threadinfo ffff880193b42000, task ffff880198af9f90)
Stack:
 0000000000000000 ffff880193b43e1c 0000000000000000 0000000000000019
 ffff880193b43c08 ffff880100000000 ffff88017af80180 ffff880100000000
 ffff880173533400 ffff880198af9f90 000000009fc91540 ffff88017af801b0
Call Trace:
 [<ffffffff810e9d74>] __split_huge_page_pmd+0xe4/0x280
 [<ffffffff810a9b9e>] ? free_hot_cold_page_list+0x3e/0x60
 [<ffffffff810c22cd>] unmap_single_vma+0x77d/0x820
 [<ffffffff810c2c14>] zap_page_range+0xa4/0xe0
 [<ffffffff813d9846>] ? sys_recvfrom+0xd6/0x120
 [<ffffffff810bfa7d>] sys_madvise+0x31d/0x660
 [<ffffffff81482b2d>] system_call_fastpath+0x1a/0x1f
Code: 83 39 00 f3 90 49 8b 45 00 a9 00 00 80 00 75 f3 41 ff 84 24 44
e0 ff ff f0 41 0f ba 6d 00 17 19 c0 85 c0 0f 84 d7 fa ff ff eb c8 <0f>
0b 8b 53 18 8b 75 9c ff c2 48 c7 c7 60 95 5c 81 31 c0 e8 ac
RIP  [<ffffffff810e89c9>] split_huge_page+0x739/0x7a0
 RSP <ffff880193b43b78>
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [hidden email]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/
Reply | Threaded
Open this post in threaded view
|

Re: kernel BUG at mm/huge_memory.c:1798!

Hillf Danton
On Thu, Dec 27, 2012 at 8:31 AM, Alexander Beregalov
<[hidden email]> wrote:

> On 25 December 2012 16:05, Hillf Danton <[hidden email]> wrote:
>> On Tue, Dec 25, 2012 at 12:38 PM, Zhouping Liu <[hidden email]> wrote:
>>> Hello all,
>>>
>>> I found the below kernel bug using latest mainline(637704cbc95),
>>> my hardware has 2 numa nodes, and it's easy to reproduce the issue
>>> using LTP test case: "# ./mmap10 -a -s -c 200":
>>
>> Can you test with 5a505085f0 and 4fc3f1d66b1 reverted?
>>
>
> Hello,
> does it look like the same problem?

Yes, it is, and thank you for reporting the oops.

Hillf

[  588.143072] mapcount 0 page_mapcount 3
[  588.147471] ------------[ cut here ]------------
[  588.152856] kernel BUG at mm/huge_memory.c:1798!
>
> mapcount 0 page_mapcount 1
> ------------[ cut here ]------------
> kernel BUG at mm/huge_memory.c:1798!


> invalid opcode: 0000 [#1] PREEMPT SMP
> Modules linked in: r8169 radeon cfbfillrect cfbimgblt cfbcopyarea
> i2c_algo_bit backlight drm_kms_helper ttm drm agpgart
> CPU 3
> Pid: 15825, comm: firefox Not tainted 3.8.0-rc1-00004-g637704c #1
> Gigabyte Technology Co., Ltd. P35-DS3/P35-DS3
> RIP: 0010:[<ffffffff810e89c9>]  [<ffffffff810e89c9>] split_huge_page+0x739/0x7a0
> RSP: 0018:ffff880193b43b78  EFLAGS: 00010297
> RAX: 0000000000000001 RBX: ffffea0002fd0000 RCX: ffffffff8175e078
> RDX: 000000000000003e RSI: ffffea0002fd0000 RDI: 0000000000000246
> RBP: ffff880193b43c48 R08: 000000000000ffff R09: 0000000000000000
> R10: 00000000000002d5 R11: 0000000000000000 R12: 0000000000000000
> R13: ffff880173533464 R14: 00007f0973000000 R15: ffffea0002fd0000
> FS:  00007f09b8db6740(0000) GS:ffff88019fd80000(0000) knlGS:0000000000000000
> CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> CR2: 00007ff210e78008 CR3: 0000000195379000 CR4: 00000000000007e0
> DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> Process firefox (pid: 15825, threadinfo ffff880193b42000, task ffff880198af9f90)
> Stack:
>  0000000000000000 ffff880193b43e1c 0000000000000000 0000000000000019
>  ffff880193b43c08 ffff880100000000 ffff88017af80180 ffff880100000000
>  ffff880173533400 ffff880198af9f90 000000009fc91540 ffff88017af801b0
> Call Trace:
>  [<ffffffff810e9d74>] __split_huge_page_pmd+0xe4/0x280
>  [<ffffffff810a9b9e>] ? free_hot_cold_page_list+0x3e/0x60
>  [<ffffffff810c22cd>] unmap_single_vma+0x77d/0x820
>  [<ffffffff810c2c14>] zap_page_range+0xa4/0xe0
>  [<ffffffff813d9846>] ? sys_recvfrom+0xd6/0x120
>  [<ffffffff810bfa7d>] sys_madvise+0x31d/0x660
>  [<ffffffff81482b2d>] system_call_fastpath+0x1a/0x1f
> Code: 83 39 00 f3 90 49 8b 45 00 a9 00 00 80 00 75 f3 41 ff 84 24 44
> e0 ff ff f0 41 0f ba 6d 00 17 19 c0 85 c0 0f 84 d7 fa ff ff eb c8 <0f>
> 0b 8b 53 18 8b 75 9c ff c2 48 c7 c7 60 95 5c 81 31 c0 e8 ac
> RIP  [<ffffffff810e89c9>] split_huge_page+0x739/0x7a0
>  RSP <ffff880193b43b78>
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [hidden email]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/
Reply | Threaded
Open this post in threaded view
|

Re: BUG: unable to handle kernel NULL pointer dereference at 0000000000000500

Zlatko Calusic
In reply to this post by Zhouping Liu
On 26.12.2012 12:22, Zhouping Liu wrote:

> Hello everyone,
>
> The latest mainline(637704cbc95c) would trigger the following error when the system was under
> some pressure condition(in my testing, I used oom01 case inside LTP test suite to trigger the issue):
>
> [ 5462.920151] BUG: unable to handle kernel NULL pointer dereference at 0000000000000500
> [ 5462.927991] IP: [<ffffffff811542d9>] wait_iff_congested+0x59/0x140
> [ 5462.934176] PGD 0
> [ 5462.936191] Oops: 0000 [#2] SMP
> [ 5462.939428] Modules linked in: lockd sunrpc iptable_mangle ipt_REJECT nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_tabled
> [ 5462.984261] CPU 13
> [ 5462.986184] Pid: 117, comm: kswapd3 Tainted: G      D      3.8.0-rc1+ #1 Dell Inc. PowerEdge M905/0D413F
> [ 5462.995814] RIP: 0010:[<ffffffff811542d9>]  [<ffffffff811542d9>] wait_iff_congested+0x59/0x140
> [ 5463.004411] RSP: 0018:ffff88007c97fd48  EFLAGS: 00010202
> [ 5463.009701] RAX: 0000000000000001 RBX: 0000000000000064 RCX: 0000000000000001
> [ 5463.016818] RDX: 0000000000000064 RSI: 0000000000000000 RDI: 0000000000000000
> [ 5463.023926] RBP: ffff88007c97fd98 R08: 0000000000000000 R09: ffff88022ffd9d80
> [ 5463.031033] R10: 0000000000003189 R11: 0000000000000000 R12: 00000001004ee87e
> [ 5463.038140] R13: 0000000000000002 R14: 0000000000000000 R15: ffff88022ffd9000
> [ 5463.045258] FS:  00007f3e570de740(0000) GS:ffff88022fcc0000(0000) knlGS:0000000000000000
> [ 5463.053317] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [ 5463.059041] CR2: 0000000000000500 CR3: 00000000018dc000 CR4: 00000000000007e0
> [ 5463.066157] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> [ 5463.073276] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> [ 5463.080400] Process kswapd3 (pid: 117, threadinfo ffff88007c97e000, task ffff88007c981970)
> [ 5463.088633] Stack:
> [ 5463.090646]  ffff88007c97fd98 0000000000000000 ffff88007c981970 ffffffff81086080
> [ 5463.098090]  ffff88007c97fd68 ffff88007c97fd68 ffff88022ffd9d80 0000000000000002
> [ 5463.105527]  0000000000000002 0000000000000000 ffff88007c97feb8 ffffffff8114b0e3
> [ 5463.112998] Call Trace:
> [ 5463.115446]  [<ffffffff81086080>] ? wake_up_bit+0x40/0x40
> [ 5463.120826]  [<ffffffff8114b0e3>] kswapd+0x6c3/0xa50
> [ 5463.125775]  [<ffffffff8114aa20>] ? zone_reclaim+0x270/0x270
> [ 5463.131415]  [<ffffffff81085680>] kthread+0xc0/0xd0
> [ 5463.136278]  [<ffffffff810855c0>] ? kthread_create_on_node+0x120/0x120
> [ 5463.142786]  [<ffffffff8160a0ac>] ret_from_fork+0x7c/0xb0
> [ 5463.148166]  [<ffffffff810855c0>] ? kthread_create_on_node+0x120/0x120
> [ 5463.154668] Code: 4e 6d 88 00 48 c7 45 b8 00 00 00 00 48 83 c0 18 48 c7 45 c8 80 60 08 81 48 89 45 d0 48 89 45 d8 8b 04 b5 a0 9a cd 81 85 c0 74 0f <48> 8b 87 00 05 00 00 a8 04 0f 85 98 00 00 00 e8 b3 c3
> [ 5463.174097] RIP  [<ffffffff811542d9>] wait_iff_congested+0x59/0x140
> [ 5463.180352]  RSP <ffff88007c97fd48>
> [ 5463.183824] CR2: 0000000000000500
> [ 5463.203717] ---[ end trace 9ff4ff9087c13a36 ]---
>
> I attached the config file, hope it can make some help.
>
> Thanks,
> Zhouping
>

Thank you for the report Zhouping!

Would you be so kind to test the following patch and report results? Apply the patch to the latest mainline.

Thanks,

diff --git a/mm/vmscan.c b/mm/vmscan.c
index 23291b9..e55ce55 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -2564,6 +2564,7 @@ static bool prepare_kswapd_sleep(pg_data_t *pgdat, int order, long remaining,
 static unsigned long balance_pgdat(pg_data_t *pgdat, int order,
  int *classzone_idx)
 {
+ bool pgdat_is_balanced = false;
  struct zone *unbalanced_zone;
  int i;
  int end_zone = 0; /* Inclusive.  0 = ZONE_DMA */
@@ -2638,8 +2639,11 @@ loop_again:
  zone_clear_flag(zone, ZONE_CONGESTED);
  }
  }
- if (i < 0)
+
+ if (i < 0) {
+ pgdat_is_balanced = true;
  goto out;
+ }
 
  for (i = 0; i <= end_zone; i++) {
  struct zone *zone = pgdat->node_zones + i;
@@ -2766,8 +2770,11 @@ loop_again:
  pfmemalloc_watermark_ok(pgdat))
  wake_up(&pgdat->pfmemalloc_wait);
 
- if (pgdat_balanced(pgdat, order, *classzone_idx))
+ if (pgdat_balanced(pgdat, order, *classzone_idx)) {
+ pgdat_is_balanced = true;
  break; /* kswapd: all done */
+ }
+
  /*
  * OK, kswapd is getting into trouble.  Take a nap, then take
  * another pass across the zones.
@@ -2775,7 +2782,7 @@ loop_again:
  if (total_scanned && (sc.priority < DEF_PRIORITY - 2)) {
  if (has_under_min_watermark_zone)
  count_vm_event(KSWAPD_SKIP_CONGESTION_WAIT);
- else
+ else if (unbalanced_zone)
  wait_iff_congested(unbalanced_zone, BLK_RW_ASYNC, HZ/10);
  }
 
@@ -2788,9 +2795,9 @@ loop_again:
  if (sc.nr_reclaimed >= SWAP_CLUSTER_MAX)
  break;
  } while (--sc.priority >= 0);
-out:
 
- if (!pgdat_balanced(pgdat, order, *classzone_idx)) {
+out:
+ if (!pgdat_is_balanced) {
  cond_resched();
 
  try_to_freeze();

--
Zlatko
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [hidden email]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/
Reply | Threaded
Open this post in threaded view
|

Re: kernel BUG at mm/huge_memory.c:1798!

Alex Xu
In reply to this post by Hillf Danton
On 25/12/12 07:05 AM, Hillf Danton wrote:

> On Tue, Dec 25, 2012 at 12:38 PM, Zhouping Liu <[hidden email]> wrote:
>> Hello all,
>>
>> I found the below kernel bug using latest mainline(637704cbc95),
>> my hardware has 2 numa nodes, and it's easy to reproduce the issue
>> using LTP test case: "# ./mmap10 -a -s -c 200":
>
> Can you test with 5a505085f0 and 4fc3f1d66b1 reverted?
>
> Hillf
>

(for people from mailing lists, please cc me when replying)

Same thing?

mapcount 0 page_mapcount 1
------------[ cut here ]------------
kernel BUG at mm/huge_memory.c:1798!
invalid opcode: 0000 [#1] SMP
Modules linked in: usb_storage wacom
CPU 3
Pid: 1287, comm: thunderbird Not tainted 3.8.0-rc1+ #5 HP-Pavilion
AU992AA-ABL e9262f/Indio
RIP: 0010:[<ffffffff810c12c0>]  [<ffffffff810c12c0>] 0xffffffff810c12c0
RSP: 0018:ffff880216887c58  EFLAGS: 00010297
RAX: 0000000000000001 RBX: ffffea00065d0000 RCX: 0000000000000038
RDX: 000000000000002c RSI: 0000000000000046 RDI: ffffffff818e9e34
RBP: ffff880216887cd8 R08: 000000000000000a R09: 0000000000000000
R10: 0000000000000272 R11: 0000000000000271 R12: ffff880203fbad10
R13: 00007f1160e00000 R14: 0000000000000000 R15: ffffea00065d0000
FS:  00007f1180fca740(0000) GS:ffff88022fd80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fca86e18000 CR3: 00000002168e3000 CR4: 00000000000007e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process thunderbird (pid: 1287, threadinfo ffff880216886000, task
ffff8802259c4380)
Stack:
 ffff880216887c68 ffffffff8145f284 ffff880216887cb8 ffff8801f9c24ac0
 00000000065d0000 ffff8801f9c24af0 0000000000000000 ffff880216887cf8
 0000000000000000 00000007f1160e00 0000000000000000 ffffea00065d0000
Call Trace:
 [<ffffffff8145f284>] ? 0xffffffff8145f284
 [<ffffffff810c2310>] 0xffffffff810c2310
 [<ffffffff810a69ea>] 0xffffffff810a69ea
 [<ffffffff8106bbd8>] ? 0xffffffff8106bbd8
 [<ffffffff810a7710>] 0xffffffff810a7710
 [<ffffffff810a4a1d>] 0xffffffff810a4a1d
 [<ffffffff8106e062>] ? 0xffffffff8106e062
 [<ffffffff810adf84>] ? 0xffffffff810adf84
 [<ffffffff81460952>] 0xffffffff81460952
Code: 6d 81 31 c0 8b 75 c4 83 c2 01 e8 c4 94 39 00 e9 83 fb ff ff 0f 0b
0f 0b 0f 0b f3 90 48 8b 03 a9 00 00 80 00 75 f4 e9 b6 fb ff ff <0f> 0b
0f 0b 66 66 66 2e 0f 1f 84 00 00 00 00 00 55 49 89 d1 48
RIP  [<ffffffff810c12c0>] 0xffffffff810c12c0
 RSP <ffff880216887c58>
---[ end trace 0d442da0022ecdd1 ]---

I don't have KALLSYMS, so after running through ksymoops:

>>RIP; ffffffff810c12c0 <split_huge_page+600/610>   <=====

>>RBX; ffffea00065d0000 <phys_startup_64+ffffea00055d0000/ffffffff80000000>
>>RDI; ffffffff818e9e34 <logbuf_lock+0/4>
>>RBP; ffff880216887cd8 <phys_startup_64+ffff880215887cd8/ffffffff80000000>
>>R12; ffff880203fbad10 <phys_startup_64+ffff880202fbad10/ffffffff80000000>
>>R13; 00007f1160e00000 <phys_startup_64+7f115fe00000/ffffffff80000000>
>>R15; ffffea00065d0000 <phys_startup_64+ffffea00055d0000/ffffffff80000000>

Trace; ffffffff8145f284 <schedule+24/70>
Trace; ffffffff810c2310 <__split_huge_page_pmd+b0/1b0>
Trace; ffffffff810a69ea <unmap_single_vma+25a/700>
Trace; ffffffff8106bbd8 <futex_wake+108/130>
Trace; ffffffff810a7710 <zap_page_range+90/d0>
Trace; ffffffff810a4a1d <sys_madvise+25d/690>
Trace; ffffffff8106e062 <sys_futex+142/1a0>
Trace; ffffffff810adf84 <vm_munmap+54/70>
Trace; ffffffff81460952 <system_call_fastpath+16/1b>

Code;  ffffffff810c1295 <split_huge_page+5d5/610>
0000000000000000 <_RIP>:
Code;  ffffffff810c1295 <split_huge_page+5d5/610>
   0:   6d                        insl   (%dx),%es:(%rdi)
Code;  ffffffff810c1296 <split_huge_page+5d6/610>
   1:   81 31 c0 8b 75 c4         xorl   $0xc4758bc0,(%rcx)
Code;  ffffffff810c129c <split_huge_page+5dc/610>
   7:   83 c2 01                  add    $0x1,%edx
Code;  ffffffff810c129f <split_huge_page+5df/610>
   a:   e8 c4 94 39 00            callq  3994d3 <_RIP+0x3994d3>
Code;  ffffffff810c12a4 <split_huge_page+5e4/610>
   f:   e9 83 fb ff ff            jmpq   fffffffffffffb97
<_RIP+0xfffffffffffffb97>
Code;  ffffffff810c12a9 <split_huge_page+5e9/610>
  14:   0f 0b                     ud2
Code;  ffffffff810c12ab <split_huge_page+5eb/610>
  16:   0f 0b                     ud2
Code;  ffffffff810c12ad <split_huge_page+5ed/610>
  18:   0f 0b                     ud2
Code;  ffffffff810c12af <split_huge_page+5ef/610>
  1a:   f3 90                     pause
Code;  ffffffff810c12b1 <split_huge_page+5f1/610>
  1c:   48 8b 03                  mov    (%rbx),%rax
Code;  ffffffff810c12b4 <split_huge_page+5f4/610>
  1f:   a9 00 00 80 00            test   $0x800000,%eax
Code;  ffffffff810c12b9 <split_huge_page+5f9/610>
  24:   75 f4                     jne    1a <_RIP+0x1a>
Code;  ffffffff810c12bb <split_huge_page+5fb/610>
  26:   e9 b6 fb ff ff            jmpq   fffffffffffffbe1
<_RIP+0xfffffffffffffbe1>
Code;  ffffffff810c12c0 <split_huge_page+600/610>   <=====
  2b:   0f 0b                     ud2       <=====
Code;  ffffffff810c12c2 <split_huge_page+602/610>
  2d:   0f 0b                     ud2
Code;  ffffffff810c12c4 <split_huge_page+604/610>
  2f:   66 66 66 2e 0f 1f 84      data32 data32 nopw %cs:0x0(%rax,%rax,1)
Code;  ffffffff810c12cb <split_huge_page+60b/610>
  36:   00 00 00 00 00
Code;  ffffffff810c12d0 <do_huge_pmd_wp_page+0/840>
  3b:   55                        push   %rbp
Code;  ffffffff810c12d1 <do_huge_pmd_wp_page+1/840>
  3c:   49 89 d1                  mov    %rdx,%r9
Code;  ffffffff810c12d4 <do_huge_pmd_wp_page+4/840>
  3f:   48                        rex.W
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [hidden email]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/
Reply | Threaded
Open this post in threaded view
|

Re: BUG: unable to handle kernel NULL pointer dereference at 0000000000000500

Zlatko Calusic
In reply to this post by Zhouping Liu
On 28.12.2012 00:30, Sedat Dilek wrote:

> Hi Zlatko,
>
> I am not sure if I hit the same problem as described in this thread.
>
> Under heavy load, while building a customized toolchain for the Freetz
> router project I got a BUG || NULL pointer derefence || kswapd ||
> zone_balanced || pgdat_balanced() etc. (details see my screenshot).
>
> I will try your patch from [1] ***only*** on top of my last
> Linux-v3.8-rc1 GIT setup (post-v3.8-rc1 mainline + some net-fixes).
>

Yes, that's the same bug. It should be fixed with my latest patch, so
I'd appreciate you testing it, to be on the safe side this time. There
should be no difference if you apply it to anything newer than 3.8-rc1,
so go for it. Thanks!

Regards,
--
Zlatko
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [hidden email]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/
Reply | Threaded
Open this post in threaded view
|

Re: BUG: unable to handle kernel NULL pointer dereference at 0000000000000500

Sedat Dilek-2
On Fri, Dec 28, 2012 at 12:39 AM, Zlatko Calusic
<[hidden email]> wrote:

> On 28.12.2012 00:30, Sedat Dilek wrote:
>>
>> Hi Zlatko,
>>
>> I am not sure if I hit the same problem as described in this thread.
>>
>> Under heavy load, while building a customized toolchain for the Freetz
>> router project I got a BUG || NULL pointer derefence || kswapd ||
>> zone_balanced || pgdat_balanced() etc. (details see my screenshot).
>>
>> I will try your patch from [1] ***only*** on top of my last
>> Linux-v3.8-rc1 GIT setup (post-v3.8-rc1 mainline + some net-fixes).
>>
>
> Yes, that's the same bug. It should be fixed with my latest patch, so I'd
> appreciate you testing it, to be on the safe side this time. There should be
> no difference if you apply it to anything newer than 3.8-rc1, so go for it.
> Thanks!
>

Not sure how I can really reproduce this bug as one build worked fine
within my last v3.8-rc1 kernel.
I increased the parallel-make-jobs-number from "4" to "8" to stress a
bit harder.
Just building right now... and will report.

If you have any test-case (script or whatever), please let me/us know.

Thanks.

- Sedat -

> Regards,
> --
> Zlatko
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [hidden email]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/
Reply | Threaded
Open this post in threaded view
|

Re: BUG: unable to handle kernel NULL pointer dereference at 0000000000000500

Zlatko Calusic
On 28.12.2012 00:42, Sedat Dilek wrote:

> On Fri, Dec 28, 2012 at 12:39 AM, Zlatko Calusic
> <[hidden email]> wrote:
>> On 28.12.2012 00:30, Sedat Dilek wrote:
>>>
>>> Hi Zlatko,
>>>
>>> I am not sure if I hit the same problem as described in this thread.
>>>
>>> Under heavy load, while building a customized toolchain for the Freetz
>>> router project I got a BUG || NULL pointer derefence || kswapd ||
>>> zone_balanced || pgdat_balanced() etc. (details see my screenshot).
>>>
>>> I will try your patch from [1] ***only*** on top of my last
>>> Linux-v3.8-rc1 GIT setup (post-v3.8-rc1 mainline + some net-fixes).
>>>
>>
>> Yes, that's the same bug. It should be fixed with my latest patch, so I'd
>> appreciate you testing it, to be on the safe side this time. There should be
>> no difference if you apply it to anything newer than 3.8-rc1, so go for it.
>> Thanks!
>>
>
> Not sure how I can really reproduce this bug as one build worked fine
> within my last v3.8-rc1 kernel.
> I increased the parallel-make-jobs-number from "4" to "8" to stress a
> bit harder.
> Just building right now... and will report.
>
> If you have any test-case (script or whatever), please let me/us know.
>

Unfortunately not, I haven't reproduced it yet on my machines. But it
seems that bug will hit only under heavy memory pressure. When close to
OOM, or possibly with lots of writing to disk. It's also possible that
fragmentation of memory zones could provoke it, that means testing it
for a longer time.

--
Zlatko
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [hidden email]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/
Reply | Threaded
Open this post in threaded view
|

Re: BUG: unable to handle kernel NULL pointer dereference at 0000000000000500

David R. Piegdon
In reply to this post by Zlatko Calusic
Hi,

NOTE to everyone debugging this: reproduced quickly with X + firefox +
youtube (adobe flash plugin)

> Would you be so kind to test the following patch and report results?
> Apply the patch to the latest mainline.

I've had probably the same problem (dmesg below) and currently am trying
your patch applied to current mainline (101e5c7470eb7f). so far it looks
very good. (before: bug after 5-30 minutes, right now 1h and counting)

thanks!


[  105.164610] ------------[ cut here ]------------
[  105.164614] kernel BUG at mm/huge_memory.c:1798!
[  105.164617] invalid opcode: 0000 [#1] PREEMPT SMP
[  105.164621] Modules linked in: fuse sha256_generic xt_owner xt_LOG xt_limit xt_recent xt_conntrack xt_multiport iptable_mangle xt_DSCP iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack fbcon font bitblit softcursor fb fbdev hwmon_vid btrfs zlib_deflate zlib_inflate xfs libcrc32c snd_usb_audio uvcvideo snd_usbmidi_lib videobuf2_core snd_rawmidi videobuf2_vmalloc videobuf2_memops hid_kensington iTCO_wdt joydev gpio_ich iTCO_vendor_support raid1 fglrx(PO) coretemp kvm_intel kvm skge acpi_cpufreq lpc_ich serio_raw asus_atk0110 snd_hda_codec_hdmi intel_agp snd_hda_intel mperf intel_gtt processor snd_hda_codec sky2 agpgart snd_hwdep [last unloaded: iTCO_wdt]
[  105.164672] CPU 1
[  105.164677] Pid: 4091, comm: XPCOM CC Tainted: P           O 3.8.0-rc1+ #43 System manufacturer System Product Name/P5B-Deluxe
[  105.164679] RIP: 0010:[<ffffffff81120fb6>]  [<ffffffff81120fb6>] __split_huge_page+0x216/0x240
[  105.164688] RSP: 0018:ffff880091511c48  EFLAGS: 00010297
[  105.164690] RAX: 0000000000000001 RBX: ffff8800a210c000 RCX: 0000000000000042
[  105.164692] RDX: 00000000000000cb RSI: 0000000000000046 RDI: ffffffff81b28a20
[  105.164694] RBP: ffff880091511ca8 R08: 000000000000ffff R09: 0000000000000000
[  105.164696] R10: 000000000000043d R11: 0000000000000001 R12: ffff8800a2295c60
[  105.164698] R13: ffffea00021e0000 R14: 0000000000000000 R15: 00000007f5134600
[  105.164701] FS:  00007f514991e700(0000) GS:ffff8800bfc80000(0000) knlGS:0000000000000000
[  105.164703] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[  105.164705] CR2: 00007f5123bff000 CR3: 000000009531b000 CR4: 00000000000007e0
[  105.164707] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[  105.164709] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[  105.164712] Process XPCOM CC (pid: 4091, threadinfo ffff880091510000, task ffff8800953616b0)
[  105.164713] Stack:
[  105.164715]  ffff880000000000 ffff8800b9c834b0 00007f5134800000 000000008158c4a5
[  105.164719]  ffff8800a210c064 00007f5134600000 ffff880091511ca8 ffffea00021e0000
[  105.164723]  ffff8800b9c83480 ffff8800a210c000 ffff88009fdc1d18 ffff8800a210c064
[  105.164727] Call Trace:
[  105.164732]  [<ffffffff81121048>] split_huge_page+0x68/0xb0
[  105.164736]  [<ffffffff81121d48>] __split_huge_page_pmd+0x1a8/0x220
[  105.164740]  [<ffffffff810f72f6>] unmap_page_range+0x1b6/0x2d0
[  105.164744]  [<ffffffff810f746b>] unmap_single_vma+0x5b/0xe0
[  105.164747]  [<ffffffff810f7e6c>] zap_page_range+0xbc/0x120
[  105.164752]  [<ffffffff8108f556>] ? futex_wake+0x116/0x130
[  105.164756]  [<ffffffff8106e396>] ? pick_next_task_fair+0x36/0xb0
[  105.164760]  [<ffffffff810f4367>] madvise_vma+0xf7/0x140
[  105.164764]  [<ffffffff810fddc2>] ? find_vma_prev+0x12/0x60
[  105.164767]  [<ffffffff810f45ed>] sys_madvise+0x23d/0x330
[  105.164772]  [<ffffffff8158e712>] system_call_fastpath+0x16/0x1b
[  105.164774] Code: 48 89 df e8 ed 10 ff ff e9 ab fe ff ff 0f 0b 41 8b 55 18 8b 75 bc ff c2 48 c7 c7 38 0e 7d 81 31 c0 e8 13 9b 46 00 e9 15 ff ff ff <0f> 0b 41 8b 4d 18 89 da ff c1 8b 75 bc 48 c7 c7 58 0e 7d 81 31
[  105.164814] RIP  [<ffffffff81120fb6>] __split_huge_page+0x216/0x240
[  105.164818]  RSP <ffff880091511c48>
[  105.164823] ---[ end trace 00c060fd7d17a3d4 ]---

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [hidden email]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/
Reply | Threaded
Open this post in threaded view
|

Re: BUG: unable to handle kernel NULL pointer dereference at 0000000000000500

Zlatko Calusic
On 28.12.2012 00:55, David R. Piegdon wrote:

> Hi,
>
> NOTE to everyone debugging this: reproduced quickly with X + firefox +
> youtube (adobe flash plugin)
>
>> Would you be so kind to test the following patch and report results?
>> Apply the patch to the latest mainline.
>
> I've had probably the same problem (dmesg below) and currently am trying
> your patch applied to current mainline (101e5c7470eb7f). so far it looks
> very good. (before: bug after 5-30 minutes, right now 1h and counting)
>

That's good news, except the oops you've attached belongs to another
bug, it seems. :P

People report good results when applying Hillf Danton suggestion to
revert 5a505085f0 and 4fc3f1d66b1. So, if the bug reappears, you could
help testing with the same procedure.

[Cc: linux-mm list]

> thanks!
>
>
> [  105.164610] ------------[ cut here ]------------
> [  105.164614] kernel BUG at mm/huge_memory.c:1798!
> [  105.164617] invalid opcode: 0000 [#1] PREEMPT SMP
> [  105.164621] Modules linked in: fuse sha256_generic xt_owner xt_LOG xt_limit xt_recent xt_conntrack xt_multiport iptable_mangle xt_DSCP iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack fbcon font bitblit softcursor fb fbdev hwmon_vid btrfs zlib_deflate zlib_inflate xfs libcrc32c snd_usb_audio uvcvideo snd_usbmidi_lib videobuf2_core snd_rawmidi videobuf2_vmalloc videobuf2_memops hid_kensington iTCO_wdt joydev gpio_ich iTCO_vendor_support raid1 fglrx(PO) coretemp kvm_intel kvm skge acpi_cpufreq lpc_ich serio_raw asus_atk0110 snd_hda_codec_hdmi intel_agp snd_hda_intel mperf intel_gtt processor snd_hda_codec sky2 agpgart snd_hwdep [last unloaded: iTCO_wdt]
> [  105.164672] CPU 1
> [  105.164677] Pid: 4091, comm: XPCOM CC Tainted: P           O 3.8.0-rc1+ #43 System manufacturer System Product Name/P5B-Deluxe
> [  105.164679] RIP: 0010:[<ffffffff81120fb6>]  [<ffffffff81120fb6>] __split_huge_page+0x216/0x240
> [  105.164688] RSP: 0018:ffff880091511c48  EFLAGS: 00010297
> [  105.164690] RAX: 0000000000000001 RBX: ffff8800a210c000 RCX: 0000000000000042
> [  105.164692] RDX: 00000000000000cb RSI: 0000000000000046 RDI: ffffffff81b28a20
> [  105.164694] RBP: ffff880091511ca8 R08: 000000000000ffff R09: 0000000000000000
> [  105.164696] R10: 000000000000043d R11: 0000000000000001 R12: ffff8800a2295c60
> [  105.164698] R13: ffffea00021e0000 R14: 0000000000000000 R15: 00000007f5134600
> [  105.164701] FS:  00007f514991e700(0000) GS:ffff8800bfc80000(0000) knlGS:0000000000000000
> [  105.164703] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [  105.164705] CR2: 00007f5123bff000 CR3: 000000009531b000 CR4: 00000000000007e0
> [  105.164707] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> [  105.164709] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> [  105.164712] Process XPCOM CC (pid: 4091, threadinfo ffff880091510000, task ffff8800953616b0)
> [  105.164713] Stack:
> [  105.164715]  ffff880000000000 ffff8800b9c834b0 00007f5134800000 000000008158c4a5
> [  105.164719]  ffff8800a210c064 00007f5134600000 ffff880091511ca8 ffffea00021e0000
> [  105.164723]  ffff8800b9c83480 ffff8800a210c000 ffff88009fdc1d18 ffff8800a210c064
> [  105.164727] Call Trace:
> [  105.164732]  [<ffffffff81121048>] split_huge_page+0x68/0xb0
> [  105.164736]  [<ffffffff81121d48>] __split_huge_page_pmd+0x1a8/0x220
> [  105.164740]  [<ffffffff810f72f6>] unmap_page_range+0x1b6/0x2d0
> [  105.164744]  [<ffffffff810f746b>] unmap_single_vma+0x5b/0xe0
> [  105.164747]  [<ffffffff810f7e6c>] zap_page_range+0xbc/0x120
> [  105.164752]  [<ffffffff8108f556>] ? futex_wake+0x116/0x130
> [  105.164756]  [<ffffffff8106e396>] ? pick_next_task_fair+0x36/0xb0
> [  105.164760]  [<ffffffff810f4367>] madvise_vma+0xf7/0x140
> [  105.164764]  [<ffffffff810fddc2>] ? find_vma_prev+0x12/0x60
> [  105.164767]  [<ffffffff810f45ed>] sys_madvise+0x23d/0x330
> [  105.164772]  [<ffffffff8158e712>] system_call_fastpath+0x16/0x1b
> [  105.164774] Code: 48 89 df e8 ed 10 ff ff e9 ab fe ff ff 0f 0b 41 8b 55 18 8b 75 bc ff c2 48 c7 c7 38 0e 7d 81 31 c0 e8 13 9b 46 00 e9 15 ff ff ff <0f> 0b 41 8b 4d 18 89 da ff c1 8b 75 bc 48 c7 c7 58 0e 7d 81 31
> [  105.164814] RIP  [<ffffffff81120fb6>] __split_huge_page+0x216/0x240
> [  105.164818]  RSP <ffff880091511c48>
> [  105.164823] ---[ end trace 00c060fd7d17a3d4 ]---
>


--
Zlatko
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [hidden email]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/
Reply | Threaded
Open this post in threaded view
|

Re: BUG: unable to handle kernel NULL pointer dereference at 0000000000000500

Sedat Dilek-2
In reply to this post by Zlatko Calusic
On Fri, Dec 28, 2012 at 12:51 AM, Zlatko Calusic
<[hidden email]> wrote:

> On 28.12.2012 00:42, Sedat Dilek wrote:
>>
>> On Fri, Dec 28, 2012 at 12:39 AM, Zlatko Calusic
>> <[hidden email]> wrote:
>>>
>>> On 28.12.2012 00:30, Sedat Dilek wrote:
>>>>
>>>>
>>>> Hi Zlatko,
>>>>
>>>> I am not sure if I hit the same problem as described in this thread.
>>>>
>>>> Under heavy load, while building a customized toolchain for the Freetz
>>>> router project I got a BUG || NULL pointer derefence || kswapd ||
>>>> zone_balanced || pgdat_balanced() etc. (details see my screenshot).
>>>>
>>>> I will try your patch from [1] ***only*** on top of my last
>>>> Linux-v3.8-rc1 GIT setup (post-v3.8-rc1 mainline + some net-fixes).
>>>>
>>>
>>> Yes, that's the same bug. It should be fixed with my latest patch, so I'd
>>> appreciate you testing it, to be on the safe side this time. There should
>>> be
>>> no difference if you apply it to anything newer than 3.8-rc1, so go for
>>> it.
>>> Thanks!
>>>
>>
>> Not sure how I can really reproduce this bug as one build worked fine
>> within my last v3.8-rc1 kernel.
>> I increased the parallel-make-jobs-number from "4" to "8" to stress a
>> bit harder.
>> Just building right now... and will report.
>>
>> If you have any test-case (script or whatever), please let me/us know.
>>
>
> Unfortunately not, I haven't reproduced it yet on my machines. But it seems
> that bug will hit only under heavy memory pressure. When close to OOM, or
> possibly with lots of writing to disk. It's also possible that fragmentation
> of memory zones could provoke it, that means testing it for a longer time.
>

I tested successfully by doing simultaneously...
- building Freetz with 8 parallel make-jobs
- building Linux GIT with 1 make-job
- 9 tabs open in firefox
- In one tab I ran YouTube music video
- etc.

I am reading [1] and [2] where another user reports success by reverting this...

commit cda73a10eb3f493871ed39f468db50a65ebeddce
"mm: do not sleep in balance_pgdat if there's no i/o congestion"

BTW, this machine has also 4GiB RAM (Ubuntu/precise AMD64).

Feel free to add a "Reported-by/Tested-by" if you think this is a
positive report.

- Sedat -

[1] http://marc.info/?l=linux-mm&m=135652835931174&w=2
[2] http://marc.info/?l=linux-mm&m=135653399800655&w=2

> --
> Zlatko
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [hidden email]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/
Reply | Threaded
Open this post in threaded view
|

Re: BUG: unable to handle kernel NULL pointer dereference at 0000000000000500

Zlatko Calusic
On 28.12.2012 01:24, Sedat Dilek wrote:

> On Fri, Dec 28, 2012 at 12:51 AM, Zlatko Calusic
> <[hidden email]> wrote:
>> On 28.12.2012 00:42, Sedat Dilek wrote:
>>>
>>> On Fri, Dec 28, 2012 at 12:39 AM, Zlatko Calusic
>>> <[hidden email]> wrote:
>>>>
>>>> On 28.12.2012 00:30, Sedat Dilek wrote:
>>>>>
>>>>>
>>>>> Hi Zlatko,
>>>>>
>>>>> I am not sure if I hit the same problem as described in this thread.
>>>>>
>>>>> Under heavy load, while building a customized toolchain for the Freetz
>>>>> router project I got a BUG || NULL pointer derefence || kswapd ||
>>>>> zone_balanced || pgdat_balanced() etc. (details see my screenshot).
>>>>>
>>>>> I will try your patch from [1] ***only*** on top of my last
>>>>> Linux-v3.8-rc1 GIT setup (post-v3.8-rc1 mainline + some net-fixes).
>>>>>
>>>>
>>>> Yes, that's the same bug. It should be fixed with my latest patch, so I'd
>>>> appreciate you testing it, to be on the safe side this time. There should
>>>> be
>>>> no difference if you apply it to anything newer than 3.8-rc1, so go for
>>>> it.
>>>> Thanks!
>>>>
>>>
>>> Not sure how I can really reproduce this bug as one build worked fine
>>> within my last v3.8-rc1 kernel.
>>> I increased the parallel-make-jobs-number from "4" to "8" to stress a
>>> bit harder.
>>> Just building right now... and will report.
>>>
>>> If you have any test-case (script or whatever), please let me/us know.
>>>
>>
>> Unfortunately not, I haven't reproduced it yet on my machines. But it seems
>> that bug will hit only under heavy memory pressure. When close to OOM, or
>> possibly with lots of writing to disk. It's also possible that fragmentation
>> of memory zones could provoke it, that means testing it for a longer time.
>>
>
> I tested successfully by doing simultaneously...
> - building Freetz with 8 parallel make-jobs
> - building Linux GIT with 1 make-job
> - 9 tabs open in firefox
> - In one tab I ran YouTube music video
> - etc.
>
> I am reading [1] and [2] where another user reports success by reverting this...
>
> commit cda73a10eb3f493871ed39f468db50a65ebeddce
> "mm: do not sleep in balance_pgdat if there's no i/o congestion"
>
> BTW, this machine has also 4GiB RAM (Ubuntu/precise AMD64).
>
> Feel free to add a "Reported-by/Tested-by" if you think this is a
> positive report.
>

Thanks for the testing! And keep running it in case something
interesting pops up. ;)

No need to revert cda73a10eb because it fixes another bug. And the patch
you're now running fixes the new bug I introduced with a combination of
my latest 2 patches. Nah, it gets complicated... :)

But, at least I found the culprit and as soon as Linus applies the fix,
everything will be hunky dory again, at least on this front. :P

Thanks,
--
Zlatko
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [hidden email]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/
Reply | Threaded
Open this post in threaded view
|

Re: BUG: unable to handle kernel NULL pointer dereference at 0000000000000500

Sedat Dilek-2
On Fri, Dec 28, 2012 at 1:33 AM, Zlatko Calusic <[hidden email]> wrote:

> On 28.12.2012 01:24, Sedat Dilek wrote:
>>
>> On Fri, Dec 28, 2012 at 12:51 AM, Zlatko Calusic
>> <[hidden email]> wrote:
>>>
>>> On 28.12.2012 00:42, Sedat Dilek wrote:
>>>>
>>>>
>>>> On Fri, Dec 28, 2012 at 12:39 AM, Zlatko Calusic
>>>> <[hidden email]> wrote:
>>>>>
>>>>>
>>>>> On 28.12.2012 00:30, Sedat Dilek wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>> Hi Zlatko,
>>>>>>
>>>>>> I am not sure if I hit the same problem as described in this thread.
>>>>>>
>>>>>> Under heavy load, while building a customized toolchain for the Freetz
>>>>>> router project I got a BUG || NULL pointer derefence || kswapd ||
>>>>>> zone_balanced || pgdat_balanced() etc. (details see my screenshot).
>>>>>>
>>>>>> I will try your patch from [1] ***only*** on top of my last
>>>>>> Linux-v3.8-rc1 GIT setup (post-v3.8-rc1 mainline + some net-fixes).
>>>>>>
>>>>>
>>>>> Yes, that's the same bug. It should be fixed with my latest patch, so
>>>>> I'd
>>>>> appreciate you testing it, to be on the safe side this time. There
>>>>> should
>>>>> be
>>>>> no difference if you apply it to anything newer than 3.8-rc1, so go for
>>>>> it.
>>>>> Thanks!
>>>>>
>>>>
>>>> Not sure how I can really reproduce this bug as one build worked fine
>>>> within my last v3.8-rc1 kernel.
>>>> I increased the parallel-make-jobs-number from "4" to "8" to stress a
>>>> bit harder.
>>>> Just building right now... and will report.
>>>>
>>>> If you have any test-case (script or whatever), please let me/us know.
>>>>
>>>
>>> Unfortunately not, I haven't reproduced it yet on my machines. But it
>>> seems
>>> that bug will hit only under heavy memory pressure. When close to OOM, or
>>> possibly with lots of writing to disk. It's also possible that
>>> fragmentation
>>> of memory zones could provoke it, that means testing it for a longer
>>> time.
>>>
>>
>> I tested successfully by doing simultaneously...
>> - building Freetz with 8 parallel make-jobs
>> - building Linux GIT with 1 make-job
>> - 9 tabs open in firefox
>> - In one tab I ran YouTube music video
>> - etc.
>>
>> I am reading [1] and [2] where another user reports success by reverting
>> this...
>>
>> commit cda73a10eb3f493871ed39f468db50a65ebeddce
>> "mm: do not sleep in balance_pgdat if there's no i/o congestion"
>>
>> BTW, this machine has also 4GiB RAM (Ubuntu/precise AMD64).
>>
>> Feel free to add a "Reported-by/Tested-by" if you think this is a
>> positive report.
>>
>
> Thanks for the testing! And keep running it in case something interesting
> pops up. ;)
>
> No need to revert cda73a10eb because it fixes another bug. And the patch
> you're now running fixes the new bug I introduced with a combination of my
> latest 2 patches. Nah, it gets complicated... :)
>
> But, at least I found the culprit and as soon as Linus applies the fix,
> everything will be hunky dory again, at least on this front. :P
>

I am not subscribed to LKML and linux-mm,,,
Do you have a patch with a proper subject and descriptive text? URL?

- Sedat -

> Thanks,
> --
> Zlatko
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [hidden email]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/
123