anonymous user

Forums   Register   Login   Forgot your login/password?   Search

SphinxSE- filtering only attrs not working

Common forum | 1 | 2 | 3 | 4 | 5 | ... | 450 | 451 | 452 | 453 | next »» | Create new thread

GinG

Name: Gin G
Posts: 45

2009-08-06 22:56:58 | reply!

Hello,

I'm trying to do a query like this, using SphinxSE:
select * from table where query='filter=attrname,attrid1,attrid2';

This results in an error:
ERROR 1430 (HY000): There was a problem processing the query on the foreign data
  source. Data source error: searchd error: invalid or truncated request

I've also tried:
select * from table where query=';filter=attrname,attrid1,attrid2';

According to the manual, I'm supposed to be able to filter by just attributes, right?

How is this done with SphinxSE?

Thanks!
GinG

Name: Gin G
Posts: 45

to: GinG, 2009-08-07 00:47:18 | reply!

Just an update, I noticed a bug report about filters not working in rc2, in that it said
the issue was fixed in r1921. I grabbed 0.9.9 r1977 from the SVN, and compiled sphinxse
from source again, using that release. It doesn't work either, exact same error.

Thank you.
Arantor

Name: Pete Spicer
Posts: 4444

to: GinG, 2009-08-07 00:51:40 | reply!

> Just an update, I noticed a bug report about filters not working in rc2, in that it said
> the issue was fixed in r1921. I grabbed 0.9.9 r1977 from the SVN, and compiled sphinxse
> from source again, using that release. It doesn't work either, exact same error.

Unfortunately I think you're out of luck in SphinxSE here to be honest.

You can force it from the API using SPH_MATCH_FULLSCAN matching mode but I can't see that
replicated in the SphinxSE setup at all (it simply doesn't appear to be supported) - I'm
going to check and file this as a bug though.
GinG

Name: Gin G
Posts: 45

to: Arantor, 2009-08-07 00:55:36 | reply!

> > Just an update, I noticed a bug report about filters not working in rc2, in that it
> said the issue was fixed in r1921. I grabbed 0.9.9 r1977 from the SVN, and compiled
> sphinxse from source again, using that release. It doesn't work either, exact same error.
>
> Unfortunately I think you're out of luck in SphinxSE here to be honest.
>
> You can force it from the API using SPH_MATCH_FULLSCAN matching mode but I can't see that
> replicated in the SphinxSE setup at all (it simply doesn't appear to be supported) - I'm
> going to check and file this as a bug though.

I notice that in the SphinxSE/ha_sphinx.cc, there's:
        SPH_MATCH_FULLSCAN, ///< match all document IDs w/o fulltext query, apply filters
In ESphMatchMode, I'm hoping it's somehow usable..

Thanks!
Arantor

Name: Pete Spicer
Posts: 4444

to: GinG, 2009-08-07 00:59:22 | reply!

> I notice that in the SphinxSE/ha_sphinx.cc, there's:
> SPH_MATCH_FULLSCAN, ///< match all document IDs w/o fulltext query, apply filters
> In ESphMatchMode, I'm hoping it's somehow usable..

Unless you want to modify raw C++ source yourself then recompile SphinxSE... maybe. The
mode definition is there, but the ability to actually specify it is not - I did actually
check before confirming, against 0.9.8, 0.9.9-rc1 and SVN source.

I've filed a bug on it however - http://sphinxsearch.com/bugs/view.php?id=388

GinG

Name: Gin G
Posts: 45

to: Arantor, 2009-08-07 01:00:53 | reply!

> > I notice that in the SphinxSE/ha_sphinx.cc, there's:
> > SPH_MATCH_FULLSCAN, ///< match all document IDs w/o fulltext query, apply filters
> > In ESphMatchMode, I'm hoping it's somehow usable..
>
> Unless you want to modify raw C++ source yourself then recompile SphinxSE... maybe. The
> mode definition is there, but the ability to actually specify it is not - I did actually
> check before confirming, against 0.9.8, 0.9.9-rc1 and SVN source.
>
> I've filed a bug on it however - http://sphinxsearch.com/bugs/view.php?id=388

I'm going to give modifying the source a shot.

I'll let you know how it goes.

Thanks!
GinG

Name: Gin G
Posts: 45

to: GinG, 2009-08-07 01:10:43 | reply!

> > Unless you want to modify raw C++ source yourself then recompile SphinxSE... maybe. The
> mode definition is there, but the ability to actually specify it is not - I did actually
> check before confirming, against 0.9.8, 0.9.9-rc1 and SVN source.
> >
> > I've filed a bug on it however - http://sphinxsearch.com/bugs/view.php?id=388

Hello,

Filters even with a query don't seem to work, I figured I'd give that a shot before
moving on to source hacking.

select id from table where query='test test2;filter=attr,attrid';

Borks out with:
ERROR 1430 (HY000): There was a problem processing the query on the foreign data source.
Data source error: searchd error: invalid or truncated request

The query/attr combination exists.

I've also tried one word, and !filter.

Any ideas?

Thanks!
GinG

Name: Gin G
Posts: 45

to: GinG, 2009-08-07 01:42:54 | reply!

> select id from table where query='test test2;filter=attr,attrid';
>
> Borks out with:
> ERROR 1430 (HY000): There was a problem processing the query on the foreign data source.
> Data source error: searchd error: invalid or truncated request
>
> The query/attr combination exists.
>
> I've also tried one word, and !filter.

I was missing a ; at the end of the filter statement, but, adding that didn't fix
anything:
select id from table where query='test;filter=attr,attrid;';

groupby on the same attr works fine.

Thanks!
Arantor

Name: Pete Spicer
Posts: 4444

to: GinG, 2009-08-07 02:31:41 | reply!

> I was missing a ; at the end of the filter statement, but, adding that didn't fix
> anything:
> select id from table where query='test;filter=attr,attrid;';

I'm assuming that attrid is a numeric value being handed here, and that attr is a
non-float, non str2ordinal attribute?
GinG

Name: Gin G
Posts: 45

to: Arantor, 2009-08-07 02:37:14 | reply!

> I'm assuming that attrid is a numeric value being handed here, and that attr is a
> non-float, non str2ordinal attribute?

Correct on both counts, attr is a bigint, and attrid is a bigint, i.e:
attr,478397954346262111

Thanks!
Arantor

Name: Pete Spicer
Posts: 4444

to: GinG, 2009-08-07 02:39:41 | reply!

> > I'm assuming that attrid is a numeric value being handed here, and that attr is a
> non-float, non str2ordinal attribute?
>
> Correct on both counts, attr is a bigint, and attrid is a bigint, i.e:
> attr,478397954346262111

And what version of Sphinx/SphinxSE are you using currently?

I'm not sure SphinxSE was upgraded to support bigints...
GinG

Name: Gin G
Posts: 45

to: Arantor, 2009-08-07 02:52:51 | reply!

> > > I'm assuming that attrid is a numeric value being handed here, and that attr is a
> non-float, non str2ordinal attribute?
> >
> > Correct on both counts, attr is a bigint, and attrid is a bigint, i.e:
> > attr,478397954346262111
>
> And what version of Sphinx/SphinxSE are you using currently?
>
> I'm not sure SphinxSE was upgraded to support bigints...

SphinxSE outputs bigints fine for versions >0.9.8. Like I said, the groupby works fine.

My searchd is running 0.9.9-rc2.

For SphinxSE, I've tried:
0.9.9-rc2
0.9.9 from SVN (r1977)
Currently using:
0.9.9-rc1 from the downloads page.

I've recompiled MySQL everytime for each version, group bys work, a regular select works,
but as soon as i put any filters in, it gives me the error.

I'm using rc1 now because of what's stated in this bug report:
http://sphinxsearch.com/bugs/view.php?id=325

In it the person stated that rc1 worked fine, with searchd running rc2.

What do you suggest? Could you give the code a look and see if maybe filtering doesn't
support bigints for whatever reason?

Thanks!
Arantor

Name: Pete Spicer
Posts: 4444

to: GinG, 2009-08-07 02:55:55 | reply!

> I've recompiled MySQL everytime for each version, group bys work, a regular select works,
> but as soon as i put any filters in, it gives me the error.

Then perhaps it doesn't support bigints. Can you filter on a smaller value?

Note that you don't have to recompile MySQL - only the plugin.

> In it the person stated that rc1 worked fine, with searchd running rc2.

Yeah, they've used the SphinxSE build from rc1 in that situation.

> What do you suggest? Could you give the code a look and see if maybe filtering doesn't
> support bigints for whatever reason?

I'm out of ideas. It would probably help if I were a C++ programmer... but I'm not.
GinG

Name: Gin G
Posts: 45

to: Arantor, 2009-08-07 02:59:30 | reply!

I just skimmed through the code in the SVN, and noticed:

                        case SPH_FILTER_VALUES:
                                SendInt ( tFilter.m_iValues );
                                for ( int k=0; k<tFilter.m_iValues; k++ )
                                SendUint64 ( tFilter.m_pValues[k] );
                                break;

So it seems to be sending Uint64 values, so that should be fine, I think..

I'll try with smaller values tomorrow and if that works, see if I can spot anywhere else
that they're using 32bit ints, instead of 64bit.

Thanks for your help!
GinG

Name: Gin G
Posts: 45

to: GinG, 2009-08-07 03:07:22 | reply!

> I just skimmed through the code in the SVN, and noticed:
>
> case SPH_FILTER_VALUES:
> SendInt ( tFilter.m_iValues );
> for ( int k=0; k<tFilter.m_iValues; k++ )
> SendUint64 ( tFilter.m_pValues[k] );
> break;
>
> So it seems to be sending Uint64 values, so that should be fine, I think..
>
> I'll try with smaller values tomorrow and if that works, see if I can spot anywhere else
> that they're using 32bit ints, instead of 64bit.
>
> Thanks for your help!
>

I see something!

In revision 1618, (which is the closest I could get to rc1 in the SVN), its:
                        case SPH_FILTER_VALUES:
                                SendInt ( tFilter.m_iValues );
                                for ( int k=0; k<tFilter.m_iValues; k++ )
                                SendInt ( tFilter.m_pValues[k] );
                                break;

In the later revisions, that's changed to
                                SendUint64 ( tFilter.m_pValues[k] );

So, tomorrow I'm going to switch back to a later revision and try again.

Thank you for triggering the bigint idea.
shodan

Name: Andrew Aksyonoff
Posts: 4348

to: GinG, 2009-08-11 21:29:59 | reply!

> So, tomorrow I'm going to switch back to a later revision and try again.

Later as in r1979? ;)
ninsky

Name: nin
Posts: 16

to: shodan, 2011-07-20 15:00:33 | reply!

> > So, tomorrow I'm going to switch back to a later revision and try again.
>
> Later as in r1979? ;)

Ok, SphinxSE fullscans with Sphinx 2.01b seem to be possible now. But what's the syntax?
Tried the following queries which doesn't work:

where query=";mode=any";
where query=" ;mode=any";
where query="mode=any";

(same with mode = "all" or "any")

Thanks in advance
ninsky
barryhunter

Name: Barry Hunter
Posts: 5560

to: ninsky, 2011-07-20 16:57:36 | reply!

where query=";mode=fullscan;filter=group_id,1,5,19;";

I think.
ninsky

Name: nin
Posts: 16

to: barryhunter, 2011-07-20 18:02:27 | reply!

>
> where query=";mode=fullscan;filter=group_id,1,5,19;";
>
> I think.

mode=fullscan didn't work for me although the "fullscan" seems to be valid (If I try
"full" instead of "fullscan" I get an error message.)

Common forum | 1 | 2 | 3 | 4 | 5 | ... | 450 | 451 | 452 | 453 | next »» | Create new thread