MantisBT - Issues

0001546: searchd lefts pipe after each HUP

Mon, 20 May 2013 19:37:36 +0200

Searchd lefts opened pipe after each HUP received.

Sometime it chashes after reaching OS limits:
[Fri May 17 15:30:13.279 2013] [30775] caught SIGHUP (seamless=1, in queue=1)
[Fri May 17 15:30:13.303 2013] [30775] FATAL: pipe() failed (error=Too many open files)
[Fri May 17 15:30:13.309 2013] [30775] shutdown complete

Sometimes is not:
[Fri May 17 15:55:44.410 2013] [ 5286] caught SIGHUP (seamless=1, in queue=1)
[Fri May 17 15:55:44.410 2013] [ 5286] WARNING: failed to parse config file '/etc/sphinx2.conf'; using previous settings
[Fri May 17 15:55:44.410 2013] [ 5286] WARNING: rotating index 'test1': prealloc: failed to open /data/soft/sphinx2/var/data/test1.new.spd: Too many open files; using old index

0001547: Unexpectable results of search by rt_attr_bigint

Mon, 20 May 2013 18:39:18 +0200

Hello.
We are working on project, where we have few RT indexes, and each of them serve several hundred thousands documents.
We faced a problem with searching: some of documents, which satisfy search criteria, are not seen in the result set.

We use query with mix of ‘match’ and attribute equality criteria.
Index have two docs with customer_id = 16225. One of them is doc with order_number = ‘31300222723’ and id = 4611686018427656993 :

mysql> select id, customer_id from order_index where match(‘(@r_status P)(@order_number 31300222723)')
order by publish_date desc,last_event_date desc limit 0, 10;
+---------------------+-------------+
| id | customer_id |
+---------------------+-------------+
| 4611686018427656993 | 16225 |
+---------------------+-------------+
1 row in set (0.17 sec)

But when we try to fetch all documents with customer_id = 16255 , sphinx doesn’t return the mentioned doc:

mysql> select id, customer_id from order_index where match('(@r_status P)') and customer_id = 16225
order by publish_date desc,last_event_date desc limit 0, 10;

+---------------------+-------------+
| id | customer_id |
+---------------------+-------------+
| 4611686018427676230 | 16225 |
+---------------------+-------------+
1 row in set (0.20 sec)

Is it known issue?

0001434: Indexer crashes while merging two indexes

Sun, 19 May 2013 12:48:30 +0200

Indexer crashes while merging two indexes, possibly as effect of scale - indexed database is pretty big (3+ million records)

Sphinx 2.0.6-release (r3473)
Copyright (c) 2001-2012, Andrew Aksyonoff
Copyright (c) 2008-2012, Sphinx Technologies Inc (http://sphinxsearch.com [^])

using config file '/opt/piler/etc/sphinx.conf'...
merging index 'delta1' into index 'main1'...
read 209.5 of 209.5 MB, 100.0% done
*** Oops, indexer crashed! Please send the following report to developers.
Sphinx 2.0.6-release (r3473)
-------------- report begins here ---------------
Current document: docid=0, hits=0
Current batch: minid=0, maxid=0
Hit pool start: docid=0, hit=0
-------------- backtrace begins here ---------------
Program compiled with gcc 4.6
Host OS is Linux mpiler 3.2.0-35-generic #55-Ubuntu SMP Wed Dec 5 17:42:16 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux
Stack bottom = 0x0, thread stack size = 0x10000
begin of system backtrace:
begin of system symbols:
/opt/piler/bin/indexer[0x4c4919]
/opt/piler/bin/indexer[0x405cf0]
/lib/x86_64-linux-gnu/libpthread.so.0(+0xfcb0)[0x7fc659cb6cb0]
/opt/piler/bin/indexer[0x4237fd]
/opt/piler/bin/indexer[0x491de6]
/opt/piler/bin/indexer[0x451704]
/opt/piler/bin/indexer[0x40861a]
/opt/piler/bin/indexer[0x40442c]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xed)[0x7fc6588aa76d]
/opt/piler/bin/indexer[0x405b89]
Backtrace looks OK. Now you have to do following steps:
1. Run the command over the crashed binary (for example, 'indexer'):
nm -n indexer > indexer.sym
2. Attach the binary, generated .sym and the text of backtrace (see above) to the bug report.
Also you can read the section about resolving backtraces in the documentation.
-------------- backtrace ends here ---------------

0001543: Multibyte chars in utf8 in the wordfroms file is threated as two different words

Wed, 15 May 2013 10:35:15 +0200

I think something went wrong in this bugfix;
http://sphinxsearch.com/bugs/view.php?id=1496 [^]

All words with multibyte characters (ie: å ä ö) are threated as two words and throws this warning:

WARNING: invalid mapping (must be exactly 1 destination keyword) ( wordforms='öljymaissaan > öljymaissa' ). Fix your wordforms file '/usr/local/etc/203configs/sphinx/wordforms/fi_FI.txt'.

0001542: Filters by Json Attribute doesn't work in SphinxSE

Mon, 13 May 2013 19:37:23 +0200

This works in SphinxQL

select * from realtime_media where indexedmetadata.eyes='blue';

what is the equivalent in SphinxSE?

select * from sphinxmedia where
query='mode=extended2;index=realtime_media;limit=100;filter=indexedmetadata.eyes,blue';

http://sphinxsearch.com/forum/view.html?id=10990 [^]

0001523: With SphinxSearch 2.11 _sph_geodist is 0

Mon, 13 May 2013 19:34:13 +0200

Doing a query with center point and radius.
Items are returned but their distance from center is returned as 0.

0001521: RT индекс не проходит проверку indextool

Mon, 13 May 2013 13:02:54 +0200

Только что созданный и наполненный RT индекс не проходит проверку indextool:

checking RT segment 17(25)...
FAILED, unsorted MVA values (segment=17, row=0, mvaattr=1, docid=1, val[2]=18446744073709551615, val[4]=1)

Прикрепил тестовый конфиг и набор данных, на которых воспроизводится эта проблема.

0001540: sphinxsearch build for wheezy

Mon, 13 May 2013 10:00:55 +0200

Hi,

on a debian wheezy system sphinxsearch 2.0.8 is installable without a problem but depends on libmysqlclient16 from debian squeeze.

Are there any chances to get a build for debian wheezy which depends on libmysqlclient18 so we can remove libmysqlclient16?

Thanks in advance.

0001539: Searchd doesn't delete pid file

Mon, 13 May 2013 09:51:15 +0200

Sometimes searchd doesn't delete pid file after clean shutdown. 'Clean' means 'shutdown complete' appears in the log:
[Mon May 13 10:09:25.336 2013] [10570] Child process 10571 has been forked
[Mon May 13 10:09:25.337 2013] [10571] listening on all interfaces, port=3337
[Mon May 13 10:09:25.342 2013] [10571] WARNING: wordforms contain stopwords ( wordform='ве > b' ). Fix your wordforms file '/tmp/wordforms'.
[Mon May 13 10:09:25.342 2013] [10571] WARNING: wordforms contain stopwords ( wordform='е > e' ). Fix your wordforms file '/tmp/wordforms'.
[Mon May 13 10:09:25.489 2013] [10571] WARNING: index 'vacancies_attach': '/tmp/wordforms' differs from the original
[Mon May 13 10:09:25.645 2013] [10571] WARNING: index 'resume_attach': '/tmp/wordforms' differs from the original
[Mon May 13 10:09:25.776 2013] [10571] accepting connections
[Mon May 13 11:05:32.819 2013] [10571] caught SIGTERM, shutting down
[Mon May 13 11:05:35.988 2013] [10571] shutdown complete
[Mon May 13 11:05:36.023 2013] [10570] Child process 10571 has been finished, exit code 0. Watchdog finishes also. Good bye!

But, what I see in a console during shutdown:
root@search:~# ps aux | grep 'search[d]'; cat /data/soft/sphinx/var/log/searchd.pid
sphinx 10570 0.0 0.0 30548 968 ? S 10:09 0:00 /data/soft/sphinx2//bin/searchd --config /etc/sphinx.conf
sphinx 10571 4.1 3.6 1350580 895152 ? Sl 10:09 2:21 /data/soft/sphinx2//bin/searchd --config /etc/sphinx.conf
10571
root@search:~# kill `cat /data/soft/sphinx/var/log/searchd.pid`
(waiting a few minutes, while it dumps RT indexes...)
root@search:~# ps aux | grep 'search[d]'; cat /data/soft/sphinx/var/log/searchd.pid
10571

It's was never observed with ordinary disk indexes. But in a way of searchd serving RT-indexes is in a half of stops.

0001538: Warning about wordforms after upgrade to 2.0.8

Mon, 13 May 2013 08:41:55 +0200

After upgrade to 2.0.8 I'm getting warning message:

WARNING: invalid mapping (must be exactly 1 destination keyword) ( wordforms='sodai > sodo namas' ). Fix your wordforms file '/etc/sphinx/wordforms.txt'.
WARNING: invalid mapping (must be exactly 1 destination keyword) ( wordforms='sodas > sodo namas' ). Fix your wordforms file '/etc/sphinx/wordforms.txt'.
WARNING: invalid mapping (must be exactly 1 destination keyword) ( wordforms='sodu > sodo namas' ). Fix your wordforms file '/etc/sphinx/wordforms.txt'.
WARNING: invalid mapping (must be exactly 1 destination keyword) ( wordforms='naujojoje vilnioje > naujoji vilnia' ). Fix your wordforms file '/etc/sphinx/wordforms.txt'.

According to the documentation, destination wordforms can be from couple keywords.

0001537: INTERNAL ERROR: the column 'count(*)/cnt' does not present in result set schema

Sun, 12 May 2013 21:38:30 +0200

The following behaviour occurs only if compat_sphinxql_magics is turned off.

Take a look at this query: SELECT *, count(*) as cnt, 1 as t FROM index GROUP BY t;
This works perfectly when queried as a sphinxql statement.

However when I use the php sphinxapi and try to query that I get the error "INTERNAL ERROR: the column 'count(*)/cnt' does not present in result set schema"

$client = new SphinxClient();
...
$client->SetSelect("*, count(*) as cnt, 1 as t");
$client->SetGroupBy("t", SPH_GROUPBY_ATTR);
...

0001536: Indexer crashed with "block relocation: write error"

Sat, 11 May 2013 18:55:50 +0200

I'm getting the following crash when trying to rebuild a particular index:

$ nice -n20 ionice -c2 -n7 indexer --config /home/sphinx/etc/sphinx.conf --rotate people_index
Sphinx 2.0.8-id64-release (r3831)
Copyright (c) 2001-2012, Andrew Aksyonoff
Copyright (c) 2008-2012, Sphinx Technologies Inc (http://sphinxsearch.com [^])

using config file '/home/sphinx/etc/sphinx.conf'...
indexing index 'people_index'...
collected 2346978 docs, 388.7 MB
ERROR: index 'people_index': block relocation: write error: 35186301 of 74395451 bytes written.
*** Oops, indexer crashed! Please send the following report to developers.
Sphinx 2.0.8-id64-release (r3831)
-------------- report begins here ---------------
Current document: docid=11034583, hits=31853319
Current batch: minid=0, maxid=0
Hit pool start: docid=10996477, hit=36350702
-------------- backtrace begins here ---------------
Program compiled with gcc 4.4.6
Host OS is Linux unixdeva16 2.6.32-279.19.1.el6.x86_64 0000001 SMP Sat Nov 24 14:35:28 EST 2012 x86_64 x86_64 x86_64 GNU/Linux
Stack bottom = 0x7fff10cd0450, thread stack size = 0x10000
begin of manual backtrace:
Something wrong with thread stack, backtrace may be incorrect (fp=7fff10cd0450)
Wrong stack limit or frame pointer, backtrace failed (fp=7fff10cd0450, stack=7fff10cd0000, stacksize=65536)
begin of system backtrace:

0001532: Attributes do not have values using distributed index

Tue, 07 May 2013 16:07:18 +0200

I followed the steps here in an effort to improve my sphinx search performance http://www.mysqlperformanceblog.com/2013/01/16/sphinx-search-performance-optimization-multi-threaded-search/. [^]

I query sphinx via MySQL using sphinxse.

Before distributing the indexes I receive values for my sql_attr_uint declared attributes. If I distribute the indexes using the method referenced above I do not receive the values.

With the distributed index I get:

mysql> SELECT * FROM items_sphinx WHERE items_sphinx.query='wine;index=items;mode=extended;ranker=bm25;limit=10;filter=domainid=267';
+---------+--------+-------------------------------------------------------------------------+----------+--------+---------+----------+
| id | weight | query | group_id | itemid | priceid | domainid |
+---------+--------+-------------------------------------------------------------------------+----------+--------+---------+----------+
| 9605165 | 1472 | wine;index=items;mode=extended;ranker=bm25;limit=10;filter=domainid=267 | 0 | 0 | 0 | 0 |
| 9605801 | 1472 | wine;index=items;mode=extended;ranker=bm25;limit=10;filter=domainid=267 | 0 | 0 | 0 | 0 |
| 9610040 | 1472 | wine;index=items;mode=extended;ranker=bm25;limit=10;filter=domainid=267 | 0 | 0 | 0 | 0 |
| 9610678 | 1472 | wine;index=items;mode=extended;ranker=bm25;limit=10;filter=domainid=267 | 0 | 0 | 0 | 0 |
| 9612315 | 1472 | wine;index=items;mode=extended;ranker=bm25;limit=10;filter=domainid=267 | 0 | 0 | 0 | 0 |
| 9618036 | 1472 | wine;index=items;mode=extended;ranker=bm25;limit=10;filter=domainid=267 | 0 | 0 | 0 | 0 |
| 9619428 | 1472 | wine;index=items;mode=extended;ranker=bm25;limit=10;filter=domainid=267 | 0 | 0 | 0 | 0 |
| 9620184 | 1472 | wine;index=items;mode=extended;ranker=bm25;limit=10;filter=domainid=267 | 0 | 0 | 0 | 0 |
| 9620332 | 1472 | wine;index=items;mode=extended;ranker=bm25;limit=10;filter=domainid=267 | 0 | 0 | 0 | 0 |
| 9621213 | 1472 | wine;index=items;mode=extended;ranker=bm25;limit=10;filter=domainid=267 | 0 | 0 | 0 | 0 |
+---------+--------+-------------------------------------------------------------------------+----------+--------+---------+----------+
10 rows in set (0.02 sec)

Without the distributed index I get:

mysql> SELECT * FROM items_sphinx WHERE items_sphinx.query='wine;index=items;mode=extended;ranker=bm25;limit=10;filter=domainid=267';
+---------+--------+-------------------------------------------------------------------------+----------+--------+---------+----------+
| id | weight | query | group_id | itemid | priceid | domainid |
+---------+--------+-------------------------------------------------------------------------+----------+--------+---------+----------+
| 9605165 | 1475 | wine;index=items;mode=extended;ranker=bm25;limit=10;filter=domainid=267 | 0 | 415588 | 9605165 | 267 |
| 9605801 | 1475 | wine;index=items;mode=extended;ranker=bm25;limit=10;filter=domainid=267 | 0 | 182915 | 9605801 | 267 |
| 9610040 | 1475 | wine;index=items;mode=extended;ranker=bm25;limit=10;filter=domainid=267 | 0 | 131985 | 9610040 | 267 |
| 9610678 | 1475 | wine;index=items;mode=extended;ranker=bm25;limit=10;filter=domainid=267 | 0 | 152776 | 9610678 | 267 |
| 9612315 | 1475 | wine;index=items;mode=extended;ranker=bm25;limit=10;filter=domainid=267 | 0 | 279931 | 9612315 | 267 |
| 9618036 | 1475 | wine;index=items;mode=extended;ranker=bm25;limit=10;filter=domainid=267 | 0 | 100415 | 9618036 | 267 |
| 9619428 | 1475 | wine;index=items;mode=extended;ranker=bm25;limit=10;filter=domainid=267 | 0 | 132263 | 9619428 | 267 |
| 9620184 | 1475 | wine;index=items;mode=extended;ranker=bm25;limit=10;filter=domainid=267 | 0 | 122806 | 9620184 | 267 |
| 9620332 | 1475 | wine;index=items;mode=extended;ranker=bm25;limit=10;filter=domainid=267 | 0 | 122801 | 9620332 | 267 |
| 9621213 | 1475 | wine;index=items;mode=extended;ranker=bm25;limit=10;filter=domainid=267 | 0 | 255879 | 9621213 | 267 |
+---------+--------+-------------------------------------------------------------------------+----------+--------+---------+----------+
10 rows in set (0.06 sec)

Note in the later case I have values for itemid, priceid and domainid. I define these attributes as follows in the source declaration of my sphinx.conf:

sql_attr_uint = domainid
sql_attr_uint = itemid
sql_attr_uint = priceid

In the case of the non-distributed index I have these declarations for my data source declaration. In the case of the distributed indexes I have these declarations for each of my four data source declarations.

0001259: Negative Assertion operator (aka NOT NEAR)

Tue, 07 May 2013 15:32:34 +0200

Basically a new operators to be able to run queries of the form "search for word X except when followed by Y"

Example:
match documents that do contain "beach", but not when its just part of "Beach
Street" phrase.

(just excluding the phrase, excludes all documents, even if they contain "beach" on its own elsewhere in the document)

In regular expression terms, would be an Assertion
http://www.php.net/manual/en/regexp.reference.assertions.php [^]
example: Beach(?! Street)

(I dont think any need for positive assertions with sphinx, only negative assertions)

0001529: enable star and morphology does not work together (RT Index)

Tue, 07 May 2013 15:26:23 +0200

Example of index:

index index_rt
{
type = rt
enable_star = 1
min_prefix_len = 2
prefix_fields = title, description
expand_keywords = 1
dict = keywords
path = /var/lib/sphinxsearch/data/index_rt

rt_field = title
rt_field = description
rt_attr_uint = publish_time

charset_type = utf-8
# morphology = stem_enru # if you open this string, 'enable_star' does not work
}

0001531: snippets do incorect highlighting in cases with using stopwords containing blend chars

Tue, 07 May 2013 08:43:04 +0200

source src2 : src
{
sql_query = SELECT 1, 'ONE', 22 gid, 'TWO' title
}

index asdf
{
source = src2
path = asdf
charset_type = utf-8
blend_mode = trim_none
blend_chars = -
wordforms = wordforms.txt
}

$ cat wordforms.txt
черно-белый > черно-белый
черно-бела > черно-белый
черно-белы > черно-белый
черно-белая > черно-белый
черно-белыми > черно-белый
черно-белее > черно-белый
черно-белые > черно-белый
черно-белых > черно-белый
черно-белом > черно-белый
черно-белую > черно-белый
черно-белой > черно-белый
черно-белою > черно-белый

mysql> call snippets ('черно-белый, черно-белая, черно-белым', 'asdf', 'черно-белый')\G
*************************** 1. row ***************************
snippet: черно-белый, черно-белая, черно-белым
1 row in set (0.00 sec)

0001527: License APIs under non-GPL license

Thu, 02 May 2013 21:10:37 +0200

The licenses used by the APIs:

libsphinxclient (C) -> LGPL
PHP/Python/Java: GPL
Ruby: Ruby License

The LGPL and Ruby License allow proprietary programs to call the library.

The GPL does not allow proprietary programs to call the library. Licensing libraries as GPL severely restricts what types of users can use that library.

Therefore, I recommend licensing the PHP/Python/Java libraries as LGPL or BSD.

Related:
http://sphinxsearch.com/forum/view.html?id=4196 [^]

> I contacted Shodan, and he gave me permission to use the PHP API file
> with the LGPL license :). He said he would probably change the license
> in the public release at some point too.

0001524: Using Workbench as the MySQL client for searchd

Mon, 29 Apr 2013 03:46:08 +0200

It would be nice to use MySQL Workbench as a GUI front-end for searchd.

When attempting to connect to searchd using Workbench it throws the following error:

/* Thu Apr 25 12:37:05.119 2013 conn 1 */ SHOW SESSION VARIABLES LIKE
'lower_case_table_names' # error=sphinxql: syntax error, unexpected LIKE, expecting $end
near 'LIKE 'lower_case_table_names''

There may be additional connection issues, but since this is a fatal error, it stops here.

Here is my forum post regarding this issue:
http://sphinxsearch.com/forum/view.html?id=10993 [^]

0001519: Sphinx crashes randomly.

Thu, 25 Apr 2013 17:50:57 +0200

I have occasional crashes of Sphinx on my server. Usually, it consumes all the RAM, swap and then gets killed by oom-killer. But that's another issue. Sometimes it just crashes. I have installed the debug version of Sphinx and saved two coredumps and backtraces. Both crashes happened over the weekend.

Here's the backtrace and related searchd.log records.

Crash 1

------- FATAL: CRASH DUMP -------
[Sat Apr 20 08:00:52.629 2013] [32153]

--- crashed SphinxAPI request dump ---

--- request dump end ---
Sphinx 2.2.1-id64-dev (r3813)
Handling signal 6
-------------- backtrace begins here ---------------
Program compiled with gcc 4.4.7
Configured with flags: '--enable-id64' '--prefix=/usr' '--with-libstemmer' '--with-debug' '--localstatedir=/var'
Host OS is Linux example.com 2.6.32-279.5.2.el6.x86_64 0000001 SMP Fri Aug 24 01:07:11 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux
Stack bottom = 0x7fd04247bdcf, thread stack size = 0x10000
Trying manual backtrace:
Something wrong with thread stack, manual backtrace may be incorrect (fp=0x7fd04247aba0)
Stack looks OK, attempting backtrace.
0x418b10
0x7fd0edbcc8a5
Something wrong in frame pointers, manual backtrace failed (fp=73189e)
Trying system backtrace:
begin of system symbols:
/usr/bin/searchd[0x59cd59]
/usr/bin/searchd(_ZN16SphCrashLogger_c11HandleCrashEi+0x392)[0x418b10]
/lib64/libpthread.so.0(+0xf500)[0x7fd0ef51b500]
/lib64/libc.so.6(gsignal+0x35)[0x7fd0edbcc8a5]
/lib64/libc.so.6(abort+0x175)[0x7fd0edbce085]
/lib64/libc.so.6(+0x2ba1e)[0x7fd0edbc5a1e]
/lib64/libc.so.6(__assert_perror_fail+0x0)[0x7fd0edbc5ae0]
/usr/bin/searchd[0x49c166]
/usr/bin/searchd[0x4ec564]
/usr/bin/searchd[0x4c04a5]
/usr/bin/searchd[0x45013e]
/usr/bin/searchd(_Z18RotationThreadFuncPv+0xe0)[0x450ccc]
/usr/bin/searchd(_Z20sphThreadProcWrapperPv+0x9c)[0x59ff2e]
/lib64/libpthread.so.0(+0x7851)[0x7fd0ef513851]
/lib64/libc.so.6(clone+0x6d)[0x7fd0edc8211d]
-------------- backtrace ends here ---------------
Backtrace looks OK. Now you have to do following steps:
1. Run the command over the crashed binary (for example, 'searchd'):
nm -n searchd > searchd.sym
2. Attach the binary, generated .sym and the text of backtrace (see above) to the bug report.
Also you can read the section about resolving backtraces in the documentation.
--- BT to source lines (depth 15): ---
[Sat Apr 20 08:00:53.356 2013] [32153] caught SIGHUP (seamless=1, in queue=10)
/home/user/sphinxsearch-read-only/src/sphinxutils.cpp:1935
/home/user/sphinxsearch-read-only/src/searchd.cpp:2103
??:0
??:0
??:0
??:0
??:0
/home/user/sphinxsearch-read-only/src/sphinx.cpp:6713
/home/user/sphinxsearch-read-only/src/sphinx.cpp:28553
/home/user/sphinxsearch-read-only/src/sphinx.cpp:15779
/home/user/sphinxsearch-read-only/src/searchd.cpp:17377
/home/user/sphinxsearch-read-only/src/searchd.cpp:17516
/home/user/sphinxsearch-read-only/src/sphinxstd.cpp:1029
??:0
??:0
--- BT to source lines finished ---
--- 14 active threads ---
thd 0, proto sphinxql, state net_read, command select
thd 1, proto sphinxql, state net_read, command select
thd 2, proto sphinxql, state net_read, command select
thd 3, proto sphinxql, state net_read, command select
thd 4, proto sphinxql, state net_read, command select
thd 5, proto sphinxql, state net_read, command select
thd 6, proto sphinxql, state net_read, command call
thd 7, proto sphinxql, state net_read, command select
thd 8, proto sphinxql, state net_read, command call
thd 9, proto sphinxql, state net_read, command select
thd 10, proto sphinxql, state net_read, command select
thd 11, proto sphinxql, state net_read, command call
thd 12, proto sphinxql, state net_read, command select
thd 13, proto sphinxapi, state query, command search
------- CRASH DUMP END -------

#0 0x00007fd0edbcc8a5 in raise () from /lib64/libc.so.6
0000001 0x00007fd0edbce085 in abort () from /lib64/libc.so.6
0000002 0x00007fd0edbc5a1e in __assert_fail_base () from /lib64/libc.so.6
0000003 0x00007fd0edbc5ae0 in __assert_fail () from /lib64/libc.so.6
0000004 0x000000000049c166 in CSphReader::SeekTo (this=0x7fd04247b300, iPos=670893, iSizeHint=-605357) at sphinx.cpp:6702
0000005 0x00000000004ec564 in CWordlist::ReadCP (this=0x7fd02c482980, tFile=..., uVersion=27, bWordDict=false, sError=...) at sphinx.cpp:28551
#6 0x00000000004c04a5 in CSphIndex_VLN::Prealloc (this=0x7fd02c482660, bMlock=false, bStripPath=false, sWarning=...) at sphinx.cpp:15779
0000007 0x000000000045013e in RotateIndexMT (sIndex=...) at searchd.cpp:17377
#8 0x0000000000450ccc in RotationThreadFunc () at searchd.cpp:17514
0000009 0x000000000059ff2e in sphThreadProcWrapper (pArg=0x753badb0) at sphinxstd.cpp:1028
0000010 0x00007fd0ef513851 in start_thread () from /lib64/libpthread.so.0
0000011 0x00007fd0edc8211d in clone () from /lib64/libc.so.6

Apr 20 08:04:35 production3 kernel: INFO: task searchd:32153 blocked for more than 120 seconds.
Apr 20 08:04:35 production3 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 20 08:04:35 production3 kernel: searchd D 0000000000000001 0 32153 32152 0x00000080
Apr 20 08:04:35 production3 kernel: ffff880100a47c98 0000000000000082 ffff880100a47c60 ffff880100a47c5c
Apr 20 08:04:35 production3 kernel: ffff880100a47c68 ffff88021fc24400 ffff880028276680 00000000000005a0
Apr 20 08:04:35 production3 kernel: ffff880105761058 ffff880100a47fd8 000000000000fb88 ffff880105761058
Apr 20 08:04:35 production3 kernel: Call Trace:
Apr 20 08:04:35 production3 kernel: [] exit_mm+0x95/0x180
Apr 20 08:04:35 production3 kernel: [] do_exit+0x15f/0x870
Apr 20 08:04:35 production3 kernel: [] ? __sigqueue_free+0x3d/0x50
Apr 20 08:04:35 production3 kernel: [] ? sched_clock_local+0x25/0x90
Apr 20 08:04:35 production3 kernel: [] ? __dequeue_signal+0x102/0x200
Apr 20 08:04:35 production3 kernel: [] do_group_exit+0x58/0xd0
Apr 20 08:04:35 production3 kernel: [] get_signal_to_deliver+0x1f6/0x460
Apr 20 08:04:35 production3 kernel: [] do_signal+0x75/0x800
Apr 20 08:04:35 production3 kernel: [] ? kvm_clock_read+0x1c/0x20
Apr 20 08:04:35 production3 kernel: [] ? kvm_clock_get_cycles+0x9/0x10
Apr 20 08:04:35 production3 kernel: [] ? poll_select_copy_remaining+0xf8/0x150
Apr 20 08:04:35 production3 kernel: [] do_notify_resume+0x90/0xc0
Apr 20 08:04:35 production3 kernel: [] int_signal+0x12/0x17

=============================================

Crash 2

------- FATAL: CRASH DUMP -------
[Sun Apr 21 15:48:12.879 2013] [22308]

--- crashed SphinxAPI request dump ---

--- request dump end ---
Sphinx 2.2.1-id64-dev (r3813)
Handling signal 6
-------------- backtrace begins here ---------------
Program compiled with gcc 4.4.7
Configured with flags: '--enable-id64' '--prefix=/usr' '--with-libstemmer' '--with-debug' '--localstatedir=/var'
Host OS is Linux example.com 2.6.32-279.5.2.el6.x86_64 0000001 SMP Fri Aug 24 01:07:11 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux
Stack bottom = 0x7f0ec093edcf, thread stack size = 0x10000
Trying manual backtrace:
Something wrong with thread stack, manual backtrace may be incorrect (fp=0x7f0ec093dba0)
Stack looks OK, attempting backtrace.
0x418b10
0x7f0f6e5858a5
Something wrong in frame pointers, manual backtrace failed (fp=73189e)
Trying system backtrace:
begin of system symbols:
/usr/bin/searchd[0x59cd59]
/usr/bin/searchd(_ZN16SphCrashLogger_c11HandleCrashEi+0x392)[0x418b10]
/lib64/libpthread.so.0(+0xf500)[0x7f0f6fed4500]
/lib64/libc.so.6(gsignal+0x35)[0x7f0f6e5858a5]
/lib64/libc.so.6(abort+0x175)[0x7f0f6e587085]
/lib64/libc.so.6(+0x2ba1e)[0x7f0f6e57ea1e]
/lib64/libc.so.6(__assert_perror_fail+0x0)[0x7f0f6e57eae0]
/usr/bin/searchd[0x49c166]
/usr/bin/searchd[0x4ec564]/usr/bin/searchd[0x4c04a5]
/usr/bin/searchd[0x45013e]
/usr/bin/searchd(_Z18RotationThreadFuncPv+0xe0)[0x450ccc]
/usr/bin/searchd(_Z20sphThreadProcWrapperPv+0x9c)[0x59ff2e]
/lib64/libpthread.so.0(+0x7851)[0x7f0f6fecc851]
/lib64/libc.so.6(clone+0x6d)[0x7f0f6e63b11d]
-------------- backtrace ends here ---------------
Backtrace looks OK. Now you have to do following steps:
1. Run the command over the crashed binary (for example, 'searchd'):
nm -n searchd > searchd.sym
2. Attach the binary, generated .sym and the text of backtrace (see above) to the bug report.
Also you can read the section about resolving backtraces in the documentation.
--- BT to source lines (depth 15): ---
/home/user/sphinxsearch-read-only/src/sphinxutils.cpp:1935
/home/user/sphinxsearch-read-only/src/searchd.cpp:2103
??:0
??:0
??:0
??:0
??:0
/home/user/sphinxsearch-read-only/src/sphinx.cpp:6713
/home/user/sphinxsearch-read-only/src/sphinx.cpp:28553
/home/user/sphinxsearch-read-only/src/sphinx.cpp:15779
/home/user/sphinxsearch-read-only/src/searchd.cpp:17377
/home/user/sphinxsearch-read-only/src/searchd.cpp:17516
/home/user/sphinxsearch-read-only/src/sphinxstd.cpp:1029
??:0
??:0
--- BT to source lines finished ---
--- 19 active threads ---
thd 0, proto sphinxql, state net_read, command select
thd 1, proto sphinxql, state net_read, command select
thd 2, proto sphinxql, state net_read, command select
thd 3, proto sphinxql, state net_read, command select
thd 4, proto sphinxql, state net_read, command select
thd 5, proto sphinxql, state net_read, command select
thd 6, proto sphinxql, state net_read, command select
thd 7, proto sphinxql, state net_read, command select
thd 8, proto sphinxql, state net_read, command select
thd 9, proto sphinxql, state net_read, command select
thd 10, proto sphinxql, state net_read, command select
thd 11, proto sphinxql, state net_read, command select
thd 12, proto sphinxql, state net_read, command select
thd 13, proto sphinxql, state net_read, command select
thd 14, proto sphinxql, state net_read, command select
thd 15, proto sphinxql, state net_read, command select
thd 16, proto sphinxql, state net_read, command select
thd 17, proto sphinxql, state net_read, command select
thd 18, proto sphinxql, state query, command select
------- CRASH DUMP END -------
[Sun Apr 21 16:13:01.034 2013] [22307] Child process 22308 has been killed with signal 6, core dumped, will be restarted

#0 0x00007f0f6e5858a5 in raise () from /lib64/libc.so.6
0000001 0x00007f0f6e587085 in abort () from /lib64/libc.so.6
0000002 0x00007f0f6e57ea1e in __assert_fail_base () from /lib64/libc.so.6
0000003 0x00007f0f6e57eae0 in __assert_fail () from /lib64/libc.so.6
0000004 0x000000000049c166 in CSphReader::SeekTo (this=0x7f0ec093e300, iPos=2720109, iSizeHint=-1343853) at sphinx.cpp:6702
0000005 0x00000000004ec564 in CWordlist::ReadCP (this=0x7f0ea68517e0, tFile=..., uVersion=27, bWordDict=false, sError=...) at sphinx.cpp:28551
#6 0x00000000004c04a5 in CSphIndex_VLN::Prealloc (this=0x7f0ea68514c0, bMlock=false, bStripPath=false, sWarning=...) at sphinx.cpp:15779
0000007 0x000000000045013e in RotateIndexMT (sIndex=...) at searchd.cpp:17377
#8 0x0000000000450ccc in RotationThreadFunc () at searchd.cpp:17514
0000009 0x000000000059ff2e in sphThreadProcWrapper (pArg=0x750d5800) at sphinxstd.cpp:1028
0000010 0x00007f0f6fecc851 in start_thread () from /lib64/libpthread.so.0
0000011 0x00007f0f6e63b11d in clone () from /lib64/libc.so.6

0001518: Wrong stats reported ?

Wed, 24 Apr 2013 20:11:59 +0200

Hi,

On a searchd busy box I noticed some weird things related to some of the stats.

query_wall is smaller than query_cpu

query_wall: 7882553.392
query_cpu: 25225537.097

As far as I understand query_wall is CPU time (assumed query_cpu) + IO time including network wait time. If that's true then the reported stats are wrong.

On other box I noticed reverse (apparently OK) stats:

query_wall: 5.051
query_cpu: 1.395

0001407: Request for index type=template

Tue, 23 Apr 2013 10:51:53 +0200

It would be good to have the index type, which is NOT connected to any source, have NO footprint on the disk (files, locks), and just defines a things like charset table, tokenizer, stopwords, etc.
Other words - the template.

First, it could be used directly for snippet generation (since we don't need any files from the real index there; only the tokenizer).

Second, it could be used in the inheritance line of real indexes defining: just inherit the index from it, assign source(s), define the file path - and thats all.

0001507: Searchd Crash

Mon, 22 Apr 2013 14:07:38 +0200

First thanks for you great help with:
http://sphinxsearch.com/bugs/view.php?id=1455 [^]
Threads made the difference and a connect timeout of 2 seconds helped to eliminate the problems.

For 3 weeks everything was fine but now searchd crashed on both slaves

New Problem:
Our setup is still the same one master and 2 agents. The master creates the indexes and rsycs them to the agents (2.0.5). Once the indexes are synced we send the SIGHUP to rotate the index.

The agent-log looks like this:
[Sun Apr 14 15:43:01.844 2013] [18984] rotating index 'user_galleries': success
[Sun Apr 14 15:43:01.885 2013] [18984] rotating index: all indexes done
[Sun Apr 14 15:44:01.467 2013] [18984] caught SIGHUP (seamless=1, in queue=1)
[Sun Apr 14 15:44:01.510 2013] [18984] rotating index 'mobile': started
[Sun Apr 14 15:44:01.594 2013] [18984] WARNING: rotating index 'mobile': rename '/home/sphinx/data/index_live_server1/mobile.mvp' to '/home/sphinx/data/index_live_server1/mobile.old.mvp' failed: No such file or directory
[Sun Apr 14 15:44:01.601 2013] [18984] rotating index 'mobile': success
[Sun Apr 14 15:44:01.613 2013] [18984] rotating index: all indexes done
[Sun Apr 14 15:45:21.592 2013] [18984] caught SIGHUP (seamless=1, in queue=1)
[Sun Apr 14 15:45:21.610 2013] [18984] rotating index 'user_galleries_translated_us': started
[Sun Apr 14 15:45:21.735 2013] [18984] WARNING: rotating index 'user_galleries_translated_us': rename '/home/sphinx/data/index_live_server1/user_galleries_translated_us.mvp' to '/home/sphinx/data/index_live_server1/user_galleries_transla
ted_us.old.mvp' failed: No such file or directory
[Sun Apr 14 15:45:21.748 2013] [18984] rotating index 'user_galleries_translated_us': success
[Sun Apr 14 15:45:21.763 2013] [18984] rotating index: all indexes done
[Sun Apr 14 15:47:01.711 2013] [18984] caught SIGHUP (seamless=1, in queue=1)
[Sun Apr 14 15:47:01.712 2013] [18984] WARNING: nothing to rotate after SIGHUP ( in queue=0 )
[Sun Apr 14 15:49:01.792 2013] [18984] caught SIGHUP (seamless=1, in queue=1)
[Sun Apr 14 15:49:01.841 2013] [18984] rotating index 'user_list': started
[Sun Apr 14 15:49:04.057 2013] [18984] WARNING: rotating index 'user_list': rename '/home/sphinx/data/index_live_server1/user_list.mvp' to '/home/sphinx/data/index_live_server1/user_list.old.mvp' failed: No such file or directory
[Sun Apr 14 15:49:04.368 2013] [18984] rotating index 'user_list': success
[Sun Apr 14 15:49:04.648 2013] [18984] rotating index: all indexes done
[Sun Apr 14 15:49:13.957 2013] [18984]
[Sun Apr 14 15:49:14.481 2013] [18983] Child process 18984 has been finished by CRASH_EXIT (exit code 2), will be restarted
[Sun Apr 14 15:49:14.481 2013] [18983] Child process 11216 has been forked
[Sun Apr 14 15:49:14.482 2013] [11216] listening on all interfaces, port=9312
[Sun Apr 14 15:49:17.873 2013] [11216] accepting connections
[Sun Apr 14 15:49:23.360 2013] [11216]
[Sun Apr 14 15:49:23.802 2013] [18983] Child process 11216 has been finished by CRASH_EXIT (exit code 2), will be restarted
[Sun Apr 14 15:49:23.802 2013] [18983] Child process 11438 has been forked
[Sun Apr 14 15:49:23.803 2013] [11438] listening on all interfaces, port=9312
[Sun Apr 14 15:49:27.236 2013] [11438] accepting connections
[Sun Apr 14 15:49:30.143 2013] [11438]
[Sun Apr 14 15:49:30.568 2013] [18983] Child process 11438 has been finished by CRASH_EXIT (exit code 2), will be restarted
[Sun Apr 14 15:49:30.568 2013] [18983] Child process 11540 has been forked
[Sun Apr 14 15:49:30.569 2013] [11540] listening on all interfaces, port=9312
[Sun Apr 14 15:49:33.981 2013] [11540] accepting connections
[Sun Apr 14 15:49:35.406 2013] [11540]
[Sun Apr 14 15:49:35.817 2013] [18983] Child process 11540 has been finished by CRASH_EXIT (exit code 2), will be restarted
[Sun Apr 14 15:49:35.817 2013] [18983] Child process 11617 has been forked
[Sun Apr 14 15:49:35.818 2013] [11617] listening on all interfaces, port=9312
[Sun Apr 14 15:49:39.241 2013] [11617] accepting connections
[Sun Apr 14 15:49:39.591 2013] [11617]
[Sun Apr 14 15:49:40.104 2013] [18983] Child process 11617 has been finished by CRASH_EXIT (exit code 2), will be restarted
[Sun Apr 14 15:49:40.104 2013] [18983] Child process 11638 has been forked
[Sun Apr 14 15:49:40.105 2013] [11638] listening on all interfaces, port=9312
[Sun Apr 14 15:49:43.483 2013] [11638] accepting connections
[Sun Apr 14 15:49:43.902 2013] [11638]
[Sun Apr 14 15:49:44.311 2013] [18983] Child process 11638 has been finished by CRASH_EXIT (exit code 2), will be restarted
[Sun Apr 14 15:49:44.311 2013] [18983] Child process 11676 has been forked
[Sun Apr 14 15:49:44.313 2013] [11676] listening on all interfaces, port=9312
[Sun Apr 14 15:49:47.717 2013] [11676] accepting connections
[Sun Apr 14 15:49:48.986 2013] [11676]
[Sun Apr 14 15:49:49.408 2013] [18983] Child process 11676 has been finished by CRASH_EXIT (exit code 2), will be restarted
[Sun Apr 14 15:49:49.408 2013] [18983] Child process 11747 has been forked
[Sun Apr 14 15:49:49.409 2013] [11747] listening on all interfaces, port=9312
[Sun Apr 14 15:49:52.842 2013] [11747] accepting connections
[Sun Apr 14 15:49:53.545 2013] [11747]
[Sun Apr 14 15:49:53.961 2013] [18983] Child process 11747 has been finished by CRASH_EXIT (exit code 2), will be restarted
[Sun Apr 14 15:49:53.961 2013] [18983] Child process 11792 has been forked
[Sun Apr 14 15:49:53.962 2013] [11792] listening on all interfaces, port=9312
[Sun Apr 14 15:49:57.393 2013] [11792] accepting connections
[Sun Apr 14 15:49:57.855 2013] [11792]
[Sun Apr 14 15:49:58.270 2013] [18983] Child process 11792 has been finished by CRASH_EXIT (exit code 2), will be restarted
[Sun Apr 14 15:49:58.270 2013] [18983] Child process 11827 has been forked
[Sun Apr 14 15:49:58.271 2013] [11827] listening on all interfaces, port=9312
[Sun Apr 14 15:50:01.683 2013] [11827] accepting connections
[Sun Apr 14 15:50:02.201 2013] [11827]
[Sun Apr 14 15:50:02.673 2013] [18983] Child process 11827 has been finished by CRASH_EXIT (exit code 2), will be restarted
[Sun Apr 14 15:50:02.673 2013] [18983] Child process 12136 has been forked
[Sun Apr 14 15:50:02.674 2013] [12136] listening on all interfaces, port=9312
.....
[Sun Apr 14 16:18:21.233 2013] [23830] Child process 23588 has been killed with signal 11, will be restarted
[Sun Apr 14 16:18:21.234 2013] [23830] Child process 23806 has been forked
[Sun Apr 14 16:18:21.234 2013] [23806] listening on all interfaces, port=9312

Then the next rotate came and everything is fine again.

So my thought is that seachd rotated a "broken" index. But as far as i now sphinx, it would cancel the precaching in case of a broken index and would use the old index again.

So I'm a bit confused and i have no idea how to reproduce that. The "broken" index is overritten so i cant check if it was really broken.

I hope the logs help to find the problem!

0001520: SetLimits documentation wrong/misleading.

Mon, 22 Apr 2013 13:07:54 +0200

The setlimits function is documented as such

Prototype: function SetLimits ( $offset, $limit, $max_matches=0, $cutoff=0 )

which while techically true, that is how its defined in the php file, is not how it behaves. IMHO, it would be more honest to say:

Prototype: function SetLimits ( $offset, $limit, $max_matches=1000, $cutoff=0 )

0001517: query_reads, query_readskb, query_readtime not been reported

Fri, 19 Apr 2013 13:14:32 +0200

Hi,

I noticed on some of servers we have running Sphinx the query_reads, query_readskb, query_readtime are always reported as 0 in SHOW STATUS.

The platform differences:

1. I see the stats (not 0) on FreeBSD + normal indexes

2. I see the stats always reported as 0 on Debian + rt indexes

In both cases the searchd has been started with iostats and cpustats options.

0001516: Warn if search term < min_word_len

Fri, 19 Apr 2013 12:54:58 +0200

searchd will obviously fail to match a search term which is less than min_word_len long. However there is no warning that such term will *never* match and in effect searchd silently ignores search terms which are less than min_word_len.

Thus a search for "cat" when min_word_len=4 returns "0 of 0 matches".

This is misleading - the user might think no documents contain the word they are looking for, whereas in reality there could be hundreds or more.

I think searchd should generate a 'warning' message. At least then I can report this to the user who can consider the impact.

So... please add a config option to searchd to generate a warning message when any search term is less than min_word_len.