[CPS-devel] nxlucene persistent transport

[Jean-Marc Orliaguet]

Julien Anguenot wrote:
>
>> when re-indexing an entire site, no problem is experienced (I actually
>> re-indexed 24 sites, and the persistent connection works like a charm).
>>     
>
> kewl !
>   

yes indeed :-)

>> however in production (i.e. in our case, with front-end servers behind a
>> load-balancer, virtual IPs, firewall rules, weird routing tables,
>> long-lasting connections that mysteriously die after a few minutes ...)
>> the persistent connection stops working after a few minutes. Eventually
>> the xmlrpc client in nuxeo.lucene.catalog.py fails to connect to the
>> lucene server with the message "Connection to %s FAILED.
>> nuxeo.lucene.catalog won't work until the connection will be possible
>> again" and the server needs to be restarted.
>>     
>
> Argh, this is strange... Is the NXLucene server stalled in the case ? If
> it's the case that's a bug... Can you check this point out please ?
>   

In fact the NXLucene server can still be used if zope is restarted, so 
rather I would say that it's the client that is getting stalled (the 
xmlrpc.py module which is now being bypassed).

But again the production environment that we have may not be optimal for 
HTTP persistent connections...

apart from that, the NXLucene server died a day ago, I'm not sure if 
this is related to the persistent connection bug on the client, or if it 
is due to a log rotation that failed, or an invalid query. I'm still 
trying to figure out. But since the lucene server was restarted it has 
worked without any problem.

>   
>> So I changed:
>>
>>    def _getProxy(self):
>>        if getattr(self, '_v_proxy', None) is None:
>>            self._v_proxy = xmlrpclib.ServerProxy(
>>                self._getServerURL(), transport=PersistentTransport())
>>
>> to:
>>
>>    def _getProxy(self):
>>        if getattr(self, '_v_proxy', None) is None:
>>            self._v_proxy = xmlrpclib.ServerProxy(
>>                self._getServerURL())
>>
>> and individual queries now work perfectly well too.
>>
>> So probably depending on the type of method called in the catalog
>> (search vs reindex) a persistent transport is better adapted in some
>> cases, or more dangerous in others. Note that I didn't experience any of
>> these issues in the development environment.
>>     
>
> I think before doing this we need to check the server status.. If this
> is stalled then we have an issue somewhere because it means a query's
> responsible of this...
>   

the problem is probably on the client, because it keeps on opening new 
connections and thinks that they are not usable a few minutes later, so 
it could be that the connection doesn't really get freed on the server 
either.

>   
>> Otherwise, everything works perfectly, it's a *huge* performance boost
>> compared to ZCatalog!
>>     
>
> eheh great ! Glad it does work fine :)
>
> You indexing 24 CPS sites in the same NXLucene ?
>
> Can you give us the OS against which you are running NXLucene and the
> gcc version please ?
>
> Cheers,
>
> 	J.
>
>   

yes, we have 24 CPS sites on the same lucene catalog with an extra 
'site_id' keyword to restrict searches. We are going to use the central 
catalog to spread local news and events between sites.. that's a total 
of 6 zope servers connected to the same lucene.

the lucene server setup is:

4 CPU 2GHz 64bits
Red Hat Enterprise Linux AS release 4
gcc-3.4.6-3 (included in RedHat)
pylucene2 :-)

Cheers,
/JM