anonymous user

Forums   Register   Login   Forgot your login/password?   Search

Empty query

Common forum | 1 | 2 | 3 | 4 | 5 | ... | 526 | 527 | 528 | 529 | next »» | Create new thread

garyis

Name: garyis
Posts: 36

2006-10-18 00:29:01 | reply!


Is it possible to enter no query and to just browse results based on group/date?

If not could it be a possible new feature. I would use mysql but I think doing it with
sphinx could be faster =)

Also looking forward to the new version. This is a great thing you're doing.

shodan

Name: Andrew Aksyonoff
Posts: 4360

to: garyis, 2006-10-18 01:45:41 | reply!


> Is it possible to enter no query and to just browse results based on group/date?

No - Sphinx is definitely not intended to be used as a replacement for SQL server.

garyis

Name: garyis
Posts: 36

to: shodan, 2006-10-18 14:01:07 | reply!


> > Is it possible to enter no query and to just browse results based on group/date?
>
> No - Sphinx is definitely not intended to be used as a replacement for SQL server.

ok. I just wanted to see if it's faster. Sorting huge tables with mysql can be quite slow.

If anyones interested I found a way to do this but it isn't very elegant. Include a word
with each record used in the index that is unlikely to ever be searched and search for
that. I used a three letter string with non-standard characters.

ilance

Name: Len
Posts: 20

to: garyis, 2006-10-19 16:06:02 | reply!


> ok. I just wanted to see if it's faster. Sorting huge tables with mysql can be quite slow.

And... Was it faster ?

garyis

Name: garyis
Posts: 36

to: ilance, 2006-10-20 01:34:33 | reply!


> > ok. I just wanted to see if it's faster. Sorting huge tables with mysql can be quite
> slow.
>
> And... Was it faster ?

Yes alot faster. Most of all it freed up mysql to do other things such updating, serving
content etc.

Bare in mind that my mysql index size way exceeds my ram so doing anything takes longer
than usual.

If you want to do exactly what I did. Add '___' AS b somewhere in your index sql query.
When you want to browse just search for ___ in boolean mode. I used '___' as it is rarely
searched for and treated as a word by sphinx. You'll have to extend it if your min size
is bigger than 3.

shodan

Name: Andrew Aksyonoff
Posts: 4360

to: garyis, 2006-10-23 05:59:44 | reply!


> ok. I just wanted to see if it's faster. Sorting huge tables with mysql can be quite slow.

Well, my own preference would be to optimize MySQL side in this case. Creating indexes
and/or special pre-calculated tables to accelerate selects should theoretically provide
more or less the same results.

dennisj

Name: Dennis Jacobfeuerborn
Posts: 10

to: shodan, 2006-10-27 15:19:11 | reply!


> > ok. I just wanted to see if it's faster. Sorting huge tables with mysql can be quite
> slow.
>
> Well, my own preference would be to optimize MySQL side in this case. Creating indexes
> and/or special pre-calculated tables to accelerate selects should theoretically provide
> more or less the same results.

Good point. The problem however is that people have to implement two completely different
search strategies depending only on whether the query is empty or not. Yes, you could
create special pre-calculated tables but in my case I already do this for sphinx so
having to do this twice makes this very maintenance heavy.

Given that Sphinx basically first searches for matches in the index and then filters and
sorts the result wouldn't it be relatively simple to make the search stage simply return
the entire index as matches? It looks like a small modification that could make the life
of Sphinx users a lot less painful.

shodan

Name: Andrew Aksyonoff
Posts: 4360

to: dennisj, 2006-10-28 16:10:45 | reply!


> wouldn't it be relatively simple to make the search stage simply return the entire index
> as matches?

Yes, it'd be rather easy.

The idea still seems somewhat strange to me, because I can't immediately think of any
usage scenario which would be significantly faster with selecting the necessary records
through Sphinx rather than MySQL. My guess is that it should be a matter of creating a
few indexes to optimize those slower SELECTs.

On the other hand, if there's so much demand for that feature, I could definitely add it
even if I don't quite understand the need :)

dennisj

Name: Dennis Jacobfeuerborn
Posts: 10

to: shodan, 2006-10-30 10:45:26 | reply!


> The idea still seems somewhat strange to me, because I can't immediately think of any
> usage scenario which would be significantly faster with selecting the necessary records
> through Sphinx rather than MySQL. My guess is that it should be a matter of creating a
> few indexes to optimize those slower SELECTs.

I agree that the direct database approach is most likely faster but in many cases the
difference in performance is not big enough to force users down that road which is
effectively what sphinx does at the moment.

When I ask sphinx the question "return the documents that match 'xyz' and have a userid
of 5 and sort them by date" then it replies "there you go" and delivers the requested
information. However when I ask "return the documents that have a userid of 5 and sort
them by date" which is a perfectly valid inquiry sphinx basically replies with "Well, I
could give you the answer but I refuse to because I think there is a more optimal way to
do this". I'd prefer it to answer this question with "Well, I don't think this is the
best way to do this but since it's a valid request that I can answer here you go".

Simply put I believe Sphinx should respond to all valid inquiries with a positive
response regardless of how optimal it thinks the inquiry is.

shodan

Name: Andrew Aksyonoff
Posts: 4360

to: dennisj, 2006-10-30 16:32:00 | reply!


> "return the documents that have a userid of 5 and sort them by date"

You're omitting that "match" part here, and Sphinx is a full-text search system, after
all, so there's implicit empty full-text query.

On an unrelated note, the question "what's the right answer to empty query" in itself
seems something like a Zen koan to me :)

> which is a perfectly valid inquiry sphinx basically replies with "Well, I could give you
> the answer but I refuse

No, this "Well I could" is what you just made up, it's far from being that simple.

It's possible that Sphinx doesn't keep a complete list of all document IDs at all.
Pre-0.9.7 versions never kept that list at all. 0.9.7 only keeps it if a) there actually
are additional attributes attached to each document b) external docinfo storage is
specified in config file.

So this means that if we decide to change the behaviour, from now on (ie. in
0.9.7-release and higher) we should explicitly store a complete list of document IDs and
attributes, even if it's not needed (!) at all for full-text searches.

OK, let's imagine for a moment that we decided to that.

Now, the "empty" query should always match the whole database, right?

What if the query was not empty before we applied stopwords?

What if the query was all whitespace?

What if the query was all whitespace but the whitespace is configured as a valid in-word
character - it's possible, and some installation are doing this to search through tags?

To summarise, my strong belief is that the behaviour which returns all records in N cases
and no records in M cases (depending on a number of factors from software version to the
query layout) is inconsistent and counterintuitive, and as such, a Bad Thing To Do.

> Simply put I believe Sphinx should respond to all valid inquiries with a positive
> response regardless of how optimal it thinks the inquiry is.

Now, in *some* cases Sphinx could technically return the answer to non-full-text query
("I know you have userd and date anyway, so give me all the IDs that have userid=5 sorted
by date").

My point is that this would no longer be full-text query; it's rather a simplified SELECT.

It seems more or less OK to have another mechanism to do such simplified SELECTs against
Sphinx's internal dataset.

However it's definitely not OK to interbreed full-text querying and simplified SQL - too
much issues, too much inconsistency.

garyis

Name: garyis
Posts: 36

to: shodan, 2006-10-31 11:08:12 | reply!


> What if the query was not empty before we applied stopwords?
>
> What if the query was all whitespace?
>
> What if the query was all whitespace but the whitespace is configured as a valid in-word
> character - it's possible, and some installation are doing this to search through tags?

It doesn't have to be done by leaving the query empty. How about an extra query mode, or
by not passing a query string at all.

shodan

Name: Andrew Aksyonoff
Posts: 4360

to: garyis, 2006-10-31 12:58:54 | reply!


> How about an extra query mode, or by not passing a query string at all.

I'm thinking of a new API call to run simple SELECT-like queries against internal
attributes database.

robert

Name: Robert Freund
Posts: 11

to: shodan, 2006-12-01 06:00:05 | reply!


This feature (sphinx returning search results on empty query string) is quite important
for us.

dennisj

Name: Dennis Jacobfeuerborn
Posts: 10

to: shodan, 2006-12-02 19:09:56 | reply!


> > How about an extra query mode, or by not passing a query string at all.
>
> I'm thinking of a new API call to run simple SELECT-like queries against internal
> attributes database.

Maybe that's not necessary. Wouldn't it be enough to allow users to specify the query as
"null"? Like calling "$sphinxclient->Query(null)" or maybe "$sphinxclient->NoQuery()".
That way you could get around the query ambiguities mentioned above, define a behavior
for this specific case and wouldn't require to create a whole new API.

shodan

Name: Andrew Aksyonoff
Posts: 4360

to: dennisj, 2006-12-02 19:32:22 | reply!


> Maybe that's not necessary. Wouldn't it be enough to allow users to specify the query as
> "null"? Like calling "$sphinxclient->Query(null)" or maybe "$sphinxclient->NoQuery()".

Well, these are completely different to me, and that's what I was talking about :)

Modifying existing $cl->Query ( $search_string, "idx1 idx2" ) call raises a lot of
questions - but adding another call seems OK to me.

So the code would look like $cl->SelectFiltered ( "idx1 idx2" ) and all the other calls
would stay just the same.

tibo

Name: Thibault Hanin
Posts: 16

to: shodan, 2007-01-25 08:06:06 | reply!


Was this feature (Sphinx returning search results on empty query string) implemented in
0.9.7-RC2 ? If not, is it planned for another version ?

sid

Name: Sid
Posts: 29

to: shodan, 2007-01-25 08:18:41 | reply!


Would love this feature.

shodan

Name: Andrew Aksyonoff
Posts: 4360

to: tibo, 2007-01-28 17:39:59 | reply!


> Was this feature (Sphinx returning search results on empty query string) implemented in
> 0.9.7-RC2 ?

Not in RC2, but planned.

ecaron

Name: Eric Caron
Posts: 8

to: shodan, 2007-08-01 14:23:56 | reply!


I too would find this useful. It simplifies the coding on our end and keeps the datasets
unified (currently people browsing have access to newer data than searchers).

eweaver

Name: Evan
Posts: 77

to: shodan, 2007-08-20 15:55:36 | reply!


> > Was this feature (Sphinx returning search results on empty query string) implemented in
> 0.9.7-RC2 ?
>
> Not in RC2, but planned.

Is this implemented in 0.98 yet? If not, when do you expect it to be ready?

Thanks

shodan

Name: Andrew Aksyonoff
Posts: 4360

to: eweaver, 2007-08-21 03:49:01 | reply!


> Is this implemented in 0.98 yet? If not, when do you expect it to be ready?

Not in the current snapshot. I'm hoping to add it until 098-release though.

markmcspadden

Name: Mark McSpadden
Posts: 2

to: shodan, 2007-11-07 17:26:33 | reply!


> > Is this implemented in 0.98 yet? If not, when do you expect it to be ready?
>
> Not in the current snapshot. I'm hoping to add it until 098-release though.

Any update on this feature?

dennisj

Name: Dennis Jacobfeuerborn
Posts: 10

to: markmcspadden, 2007-11-07 17:32:29 | reply!


> Any update on this feature?

From the news page:
...
Full scan, or in other words the long awaited “empty query” feature, means that Sphinx
will now process and return all document IDs if there's no query string specified.
...

scottbu24

Name: Scott Bullock
Posts: 2

to: dennisj, 2011-08-10 01:45:32 | reply!


Any update on how this works? I assume its implemented now 4 years later after this
thread discussion. Whenever, I supply empty search text I get back no records. I need
away to get them all and filter on all records without having to supply search text.

Thanks in advance,

Scott

barryhunter

Name: Barry Hunter
Posts: 7499

to: scottbu24, 2011-08-10 01:48:02 | reply!


>
> Any update on how this works?

Yes it works.

The index needs at least one attribute, and needs 'extern' storage.

Entering a empty query should just work, but can use SPH_MATCH_FULLSCAN to be explicit.

scottbu24

Name: Scott Bullock
Posts: 2

to: barryhunter, 2011-08-10 04:08:30 | reply!


Barry, thanks for the confirmation. Looks like I am using a client DLL that is preventing
the "" being set as the search text so Sphinx is not the guilty party. Good to know they
now support this. This is an important part of the app I am porting from SQL Server based
searches. Thanks again.

Common forum | 1 | 2 | 3 | 4 | 5 | ... | 526 | 527 | 528 | 529 | next »» | Create new thread