0000993: Searchd crashes on sphinql query

View Issue Details [ Jump to Notes ]

[ Issue History ] [ Print ]

Project

Category

View Status

Date Submitted

Last Update

0000993

Sphinx

general

public

2011-11-21 11:05

2011-11-23 10:16

Reporter

vvlladd

Assigned To

Tomat

Priority

high

Severity

crash

Reproducibility

N/A

Status

closed

Resolution

fixed

Platform

Linux

CentOS

OS Version

5.7 (final)

Product Version

2.0.2-beta

Target Version

Fixed in Version

2.0.2-dev

Summary

0000993: Searchd crashes on sphinql query

Description

Searchd crashes after running sphinxql select query. Even after restart it doesn't come back. I need to delete the bin logs in order to make it run again.

------- FATAL: CRASH DUMP -------
[Mon Nov 21 11:45:48.369 2011] [23516]

--- crashed SphinxQL request dump ---
SELECT
*
FROM
test_post_data_rt
WHERE
MATCH('vlad')
LIMIT 0, 20

--- request dump end ---
Sphinx 2.0.2-id64-beta (r3019)
-------------- backtrace begins here ---------------
Program compiled with gcc 4.1.2
Host OS is Linux rhel54x32 2.6.18-164.el5 0000001 SMP Tue Aug 18 15:51:54 EDT 2009 i686 i686 i386 GNU/Linux
Stack bottom = 0xffffffffaec9639f, thread stack size = 0x14000
begin of manual backtrace:
Stack looks OK, attempting backtrace.
begin of system backtrace:
[Mon Nov 21 11:45:48.918 2011] [23515] Child process 23516 has been killed with signal 11, will be restarted
[Mon Nov 21 11:45:48.919 2011] [23515] Child process 23523 has been forked
[Mon Nov 21 11:45:48.919 2011] [23523] listening on 192.168.2.111:9306
[Mon Nov 21 11:45:49.271 2011] [23523] binlog: replaying log /var/lib/sphinx/binlog.001

--- crashed SphinxAPI request dump ---

--- request dump end ---
Sphinx 2.0.2-id64-beta (r3019)
-------------- backtrace begins here ---------------
Program compiled with gcc 4.1.2
Host OS is Linux rhel54x32 2.6.18-164.el5 0000001 SMP Tue Aug 18 15:51:54 EDT 2009 i686 i686 i386 GNU/Linux
Stack bottom = 0xffffffffbfb2474b, thread stack size = 0x14000
begin of manual backtrace:
Something wrong with thread stack, backtrace may be incorrect (fp=ffffffffbfb22f18)
Stack looks OK, attempting backtrace.
809e464
826b2f5
828cc5c
828cf38
828d5ec
828d6e8
80e721d
80e8fe8
5c0e9c
809a831
Something wrong in frame pointers, backtrace failed (fp=0)
begin of system backtrace:
begin of system symbols:
/usr/bin/searchdrt(_Z12sphBacktraceib+0x2d1)[0x81e2ed1]
/usr/bin/searchdrt(_ZN16SphCrashLogger_c11HandleCrashEi+0x184)[0x809e464]
[0xdc6420]
/usr/bin/searchdrt[0x826b2f5]
/usr/bin/searchdrt(_ZNK10RtBinlog_c12ReplayCommitEijR14BinlogReader_c+0x75c)[0x828cc5c]
/usr/bin/searchdrt(_ZN10RtBinlog_c12ReplayBinlogERK17SmallStringHash_TIP9CSphIndexEji+0x268)[0x828cf38]
/usr/bin/searchdrt(_ZN10RtBinlog_c6ReplayERK17SmallStringHash_TIP9CSphIndexEjPFvvE+0x8c)[0x828d5ec]
/usr/bin/searchdrt(_Z15sphReplayBinlogRK17SmallStringHash_TIP9CSphIndexEjPFvvE+0x28)[0x828d6e8]
/usr/bin/searchdrt(_Z11ServiceMainiPPc+0x234d)[0x80e721d]
/usr/bin/searchdrt(main+0x68)[0x80e8fe8]
/lib/libc.so.6(__libc_start_main+0xdc)[0x5c0e9c]
/usr/bin/searchdrt(__gxx_personality_v0+0x265)[0x809a831]
-------------- backtrace ends here ---------------
--- 0 active threads ---
------- CRASH DUMP END -------
[Mon Nov 21 11:45:49.292 2011] [23515] Child process 23523 has been finished by CRASH_EXIT (exit code 2), will be restarted
[Mon Nov 21 11:45:49.293 2011] [23524] listening on 192.168.2.111:9306
[Mon Nov 21 11:45:49.293 2011] [23515] Child process 23524 has been forked
[Mon Nov 21 11:45:49.646 2011] [23524] binlog: replaying log /var/lib/sphinx/binlog.001

--- crashed SphinxAPI request dump ---

--- request dump end ---
Sphinx 2.0.2-id64-beta (r3019)
-------------- backtrace begins here ---------------
Program compiled with gcc 4.1.2
Host OS is Linux rhel54x32 2.6.18-164.el5 0000001 SMP Tue Aug 18 15:51:54 EDT 2009 i686 i686 i386 GNU/Linux
Stack bottom = 0xffffffffbfb2474b, thread stack size = 0x14000
begin of manual backtrace:
Something wrong with thread stack, backtrace may be incorrect (fp=ffffffffbfb22f18)
Stack looks OK, attempting backtrace.
809e464
826b2f5
828cc5c
828cf38
828d5ec
828d6e8
80e721d
80e8fe8
5c0e9c
809a831
Something wrong in frame pointers, backtrace failed (fp=0)
begin of system backtrace:
begin of system symbols:
/usr/bin/searchdrt(_Z12sphBacktraceib+0x2d1)[0x81e2ed1]
/usr/bin/searchdrt(_ZN16SphCrashLogger_c11HandleCrashEi+0x184)[0x809e464]
[0xdc6420]
/usr/bin/searchdrt[0x826b2f5]
/usr/bin/searchdrt(_ZNK10RtBinlog_c12ReplayCommitEijR14BinlogReader_c+0x75c)[0x828cc5c]
/usr/bin/searchdrt(_ZN10RtBinlog_c12ReplayBinlogERK17SmallStringHash_TIP9CSphIndexEji+0x268)[0x828cf38]
/usr/bin/searchdrt(_ZN10RtBinlog_c6ReplayERK17SmallStringHash_TIP9CSphIndexEjPFvvE+0x8c)[0x828d5ec]
/usr/bin/searchdrt(_Z15sphReplayBinlogRK17SmallStringHash_TIP9CSphIndexEjPFvvE+0x28)[0x828d6e8]
/usr/bin/searchdrt(_Z11ServiceMainiPPc+0x234d)[0x80e721d]
/usr/bin/searchdrt(main+0x68)[0x80e8fe8]
/lib/libc.so.6(__libc_start_main+0xdc)[0x5c0e9c]
/usr/bin/searchdrt(__gxx_personality_v0+0x265)[0x809a831]
-------------- backtrace ends here ---------------
--- 0 active threads ---
------- CRASH DUMP END -------
[Mon Nov 21 11:45:49.667 2011] [23515] Child process 23524 has been finished by CRASH_EXIT (exit code 2), will be restarted
[Mon Nov 21 11:45:49.668 2011] [23525] listening on 192.168.2.111:9306
[Mon Nov 21 11:45:49.669 2011] [23515] Child process 23525 has been forked
[Mon Nov 21 11:45:49.990 2011] [23525] binlog: replaying log /var/lib/sphinx/binlog.001

--- crashed SphinxAPI request dump ---

--- request dump end ---
Sphinx 2.0.2-id64-beta (r3019)
-------------- backtrace begins here ---------------
Program compiled with gcc 4.1.2
Host OS is Linux rhel54x32 2.6.18-164.el5 0000001 SMP Tue Aug 18 15:51:54 EDT 2009 i686 i686 i386 GNU/Linux
Stack bottom = 0xffffffffbfb2474b, thread stack size = 0x14000
begin of manual backtrace:
Something wrong with thread stack, backtrace may be incorrect (fp=ffffffffbfb22f18)
Stack looks OK, attempting backtrace.
809e464
826b2f5
828cc5c
828cf38
828d5ec
828d6e8
80e721d
80e8fe8
5c0e9c
809a831
Something wrong in frame pointers, backtrace failed (fp=0)
begin of system backtrace:
begin of system symbols:
/usr/bin/searchdrt(_Z12sphBacktraceib+0x2d1)[0x81e2ed1]
/usr/bin/searchdrt(_ZN16SphCrashLogger_c11HandleCrashEi+0x184)[0x809e464]
[0xdc6420]
/usr/bin/searchdrt[0x826b2f5]
/usr/bin/searchdrt(_ZNK10RtBinlog_c12ReplayCommitEijR14BinlogReader_c+0x75c)[0x828cc5c]
/usr/bin/searchdrt(_ZN10RtBinlog_c12ReplayBinlogERK17SmallStringHash_TIP9CSphIndexEji+0x268)[0x828cf38]
/usr/bin/searchdrt(_ZN10RtBinlog_c6ReplayERK17SmallStringHash_TIP9CSphIndexEjPFvvE+0x8c)[0x828d5ec]
/usr/bin/searchdrt(_Z15sphReplayBinlogRK17SmallStringHash_TIP9CSphIndexEjPFvvE+0x28)[0x828d6e8]
/usr/bin/searchdrt(_Z11ServiceMainiPPc+0x234d)[0x80e721d]
/usr/bin/searchdrt(main+0x68)[0x80e8fe8]
/lib/libc.so.6(__libc_start_main+0xdc)[0x5c0e9c]
/usr/bin/searchdrt(__gxx_personality_v0+0x265)[0x809a831]
-------------- backtrace ends here ---------------
--- 0 active threads ---
------- CRASH DUMP END -------

Additional Information

I have uploaded the config file.
Running on CentOS 5.7 final:
Pentium(R) Dual-Core CPU E5300 @ 2.60GHz
2GB Ram

Tags

No tags attached.

Attached Files

sphinxrt.conf [^] (14,568 bytes) 2011-11-21 11:05

indexbackup.tar.gz [^] (2,125 bytes) 2011-11-21 11:31

Relationships

Notes
(0002111) vvlladd (reporter) 2011-11-21 11:31	Added backup of real time indexes. Try running "SELECT * FROM test_tags_rt WHERE MATCH('vlad');" after import.

(0002112) vvlladd (reporter) 2011-11-21 12:18	Commenting out dict=keywords enable_start=1 min_prefix_len=2 in index "test_post_data_rt" seems to resolve the issue

(0002114) Tomat (manager) 2011-11-21 13:47	Have you remove index and repopulate it on changing these values in your config?

(0002116) vvlladd (reporter) 2011-11-21 13:50	yes. Haven't had a crash since i commented out the above options and recreated the index

(0002121) vvlladd (reporter) 2011-11-21 15:33	test data (indexes, config) were uploaded ftp://dev1.sphinxsearch.com/issue-994 [^] it's the same setup used in issue 994

(0002126) Tomat (manager) 2011-11-22 12:21	seems that you missed your binlog. you should copy binlog files ( binlog.* ) from your searchd directory

(0002127) vvlladd (reporter) 2011-11-22 12:54	uploaded again to: ftp://dev1.sphinxsearch.com/issue-993 [^] contents of indexdata.tar.gz: -rw------- sphinx/sphinx 8 2011-11-21 14:47 binlog.001 -rw------- sphinx/sphinx 8 2011-11-21 14:59 binlog.002 -rw------- sphinx/sphinx 8 2011-11-21 15:03 binlog.003 -rw------- sphinx/sphinx 0 2011-11-21 15:03 binlog.lock -rw------- sphinx/sphinx 13 2011-11-21 15:03 binlog.meta -rw------- sphinx/sphinx 0 2011-11-21 15:03 comments_rt.lock -rw------- sphinx/sphinx 0 2011-11-21 15:03 post_data_rt.lock -rw------- sphinx/sphinx 0 2011-11-21 15:03 post_hits_rt.lock -rw------- sphinx/sphinx 0 2011-11-21 15:03 posts_rt.lock -rw------- sphinx/sphinx 0 2011-11-21 15:03 post_user_data_rt.lock -rw------- sphinx/sphinx 0 2011-11-21 15:03 tags_rt.lock -rw------- sphinx/sphinx 0 2011-11-21 15:03 test_mva_rt.lock -rw------- sphinx/sphinx 0 2011-11-21 15:03 test_post_data_rt.lock -rw------- sphinx/sphinx 0 2011-11-21 15:03 test_post_hits_rt.lock -rw------- sphinx/sphinx 0 2011-11-21 15:03 test_posts_rt.lock -rw------- sphinx/sphinx 0 2011-11-21 15:03 test_post_user_data_rt.lock -rw------- sphinx/sphinx 0 2011-11-21 15:03 test_tags_rt.lock -rw------- sphinx/sphinx 0 2011-11-21 15:03 user_rel_rt.lock -rw-r--r-- root/root 4416 2011-11-18 10:14 users_com.spa -rw-r--r-- root/root 32215 2011-11-18 10:14 users_com.spd -rw-r--r-- root/root 401 2011-11-18 10:14 users_com.sph -rw-r--r-- root/root 40377 2011-11-18 10:14 users_com.spi -rw-r--r-- root/root 0 2011-11-18 10:14 users_com.spk -rw-r--r-- root/root 0 2011-11-18 10:14 users_com.spm -rw-r--r-- root/root 3681 2011-11-18 10:14 users_com.spp -rw-r--r-- root/root 1 2011-11-18 10:14 users_com.sps -rw------- sphinx/sphinx 0 2011-11-21 15:03 users_rt.lock -rw-r--r-- root/root 9632 2011-11-18 10:11 users.spa -rw-r--r-- root/root 58470 2011-11-18 10:11 users.spd -rw-r--r-- root/root 416 2011-11-18 10:11 users.sph -rw-r--r-- root/root 47545 2011-11-18 10:11 users.spi -rw-r--r-- root/root 0 2011-11-18 10:11 users.spk -rw-r--r-- root/root 0 2011-11-18 10:11 users.spm -rw-r--r-- root/root 18096 2011-11-18 10:11 users.spp -rw-r--r-- root/root 1 2011-11-18 10:11 users.sps -rw------- sphinx/sphinx 0 2011-11-21 15:03 votes_rt.lock -rw-r--r-- root/root 117686400 2011-11-21 15:01 worldcities.spa -rw-r--r-- root/root 207491568 2011-11-21 15:02 worldcities.spd -rw-r--r-- root/root 568 2011-11-21 15:02 worldcities.sph -rw-r--r-- root/root 55035793 2011-11-21 15:02 worldcities.spi -rw-r--r-- root/root 0 2011-11-21 15:01 worldcities.spk -rw-r--r-- root/root 0 2011-11-21 15:01 worldcities.spm -rw-r--r-- root/root 208561802 2011-11-21 15:02 worldcities.spp -rw-r--r-- root/root 17381359 2011-11-21 15:02 worldcities.sps

(0002130) Tomat (manager) 2011-11-23 10:00	fixed at r3024 It might be better to remove all RT data and repopulate these indexes due the issue 0000998

Issue History
Date Modified	Username	Field	Change
2011-11-21 11:05	vvlladd	New Issue
2011-11-21 11:05	vvlladd	File Added: sphinxrt.conf
2011-11-21 11:31	vvlladd	Note Added: 0002111
2011-11-21 11:31	vvlladd	File Added: indexbackup.tar.gz
2011-11-21 11:34	vvlladd	Issue Monitored: vvlladd
2011-11-21 12:18	vvlladd	Note Added: 0002112
2011-11-21 13:47	Tomat	Note Added: 0002114
2011-11-21 13:47	Tomat	Assigned To	=> Tomat
2011-11-21 13:47	Tomat	Status	new => assigned
2011-11-21 13:50	vvlladd	Note Added: 0002116
2011-11-21 15:33	vvlladd	Note Added: 0002121
2011-11-22 12:21	Tomat	Note Added: 0002126
2011-11-22 12:54	vvlladd	Note Added: 0002127
2011-11-23 10:00	Tomat	Note Added: 0002130
2011-11-23 10:00	Tomat	Status	assigned => closed
2011-11-23 10:00	Tomat	Resolution	open => fixed
2011-11-23 10:00	Tomat	Fixed in Version	=> 2.0.2-dev
2011-11-23 10:16	vvlladd	Issue End Monitor: vvlladd