- 天地维杰网

sequenceDiagram

    100.27主 ->> 100.27主 : 崩溃 13:23
    100.27主 ->> 111.19从 : 主从切换 13:23
    100.27主 ->> + 111.19从 : 请求主从同步 13:44
    111.19从 ->> 111.19从 : bgsave生成RDB 13:44
    loop BGSAVE
        111.19从 ->> + 111.19从 : 子线程bgsave生成RDB崩溃并重试
    end
    loop BGSAVE
        111.19从 ->> + 111.19从 : 手动bgsave生成RDB崩溃
    end
    111.19从 ->> 111.19从 : scan触发崩溃 15:01
    111.19从 ->> 100.27主 : 主从切换
    111.19从 ->> 111.19从 : 手动启动恢复 15:09

10.105.100.27 日志分析

=== REDIS BUG REPORT START: Cut & paste starting from here ===
12220:M 13 Jul 13:23:33.517 # Redis 3.2.3 crashed by signal: 11
12220:M 13 Jul 13:23:33.517 # Crashed running the instuction at: 0x4250bb
12220:M 13 Jul 13:23:33.517 # Accessing address: (nil)
12220:M 13 Jul 13:23:33.517 # Failed assertion: <no assertion failed> (<no file>:0)

------ STACK TRACE ------
EIP:
/app/cachecloud/redis/src/redis-server 0.0.0.0:7000 [cluster](clientsCronHandleTimeout+0x1b)[0x4250bb]

Backtrace:
/app/cachecloud/redis/src/redis-server 0.0.0.0:7000 [cluster](logStackTrace+0x29)[0x45d3d9]
/app/cachecloud/redis/src/redis-server 0.0.0.0:7000 [cluster](sigsegvHandler+0xaa)[0x45d8ca]
/lib64/libpthread.so.0(+0xf630)[0x7fa29bb9a630]
/app/cachecloud/redis/src/redis-server 0.0.0.0:7000 [cluster](clientsCronHandleTimeout+0x1b)[0x4250bb]
/app/cachecloud/redis/src/redis-server 0.0.0.0:7000 [cluster](clientsCron+0x6b)[0x4252cb]
/app/cachecloud/redis/src/redis-server 0.0.0.0:7000 [cluster](serverCron+0x192)[0x427182]
/app/cachecloud/redis/src/redis-server 0.0.0.0:7000 [cluster](aeProcessEvents+0x2b0)[0x421290]
/app/cachecloud/redis/src/redis-server 0.0.0.0:7000 [cluster](aeMain+0x2b)[0x4214ab]
/app/cachecloud/redis/src/redis-server 0.0.0.0:7000 [cluster](main+0x410)[0x41e4a0]
/lib64/libc.so.6(__libc_start_main+0xf5)[0x7fa29b7df555]
/app/cachecloud/redis/src/redis-server 0.0.0.0:7000 [cluster][0x41e715]

EIP:
/app/redis-3.2.3/src/server.c:937

Backtrace:
/app/redis-3.2.3/src/debug.c:853
/app/redis-3.2.3/src/debug.c:1002
??:0
/app/redis-3.2.3/src/server.c:937
/app/redis-3.2.3/src/server.c:1012
/app/redis-3.2.3/src/server.c:1169
/app/redis-3.2.3/src/ae.c:324
/app/redis-3.2.3/src/ae.c:452
/app/redis-3.2.3/src/server.c:4104
??:0
??:?

现象，无法生成rdb文件，无法主从同步

同事有印象是版本bug，之前在测试环境出过问题

有两篇文档提到大分片或者大的数据集会出现这种问题。

scan扫描大key时出现实例崩溃

不要使用gdb在线上调试，会阻塞进程。