Installing DBPrism 3.0.2.1.0 (in-Oracle-Lucene) on a WinXP-10gR2

Introduction

This post talks, shares experiences, about installing Marcelo Ochoa’s (MO, Marcelo Ochoa’s personal blog) in-oracle-lucene implementation into an oracle 10.2.0.3 on winxp. MO did a really great job in porting a raw lucene to an oracle database. His essential trick is actually to represent the file system layer, lucene usually lives in, within a blob based storage as well as employing the oracle odci-interface. Any further information is available in the online documentation here Lucene Domain Index.

Downloads

The latest version of the code stack is available as lucene-odi-bin-3.0.2.1.0.zip from http://sourceforge.net/projects/dbprism/files/odi/3.0.2.1.0/. Do not care about the sourceforge project being called DBPrism or this other code stack around being tagged ojvm, this is all history. The latest version to comprise the lucene 3.0.2 core base is lucene-odi, lucene-ojvm currently features the lucene 2.9.2 core base as just a maintainance release.

Another necessary download is ant, as a build and installation runtime environment. I sourced apache-ant-1.8.1-bin.zip from http://ant.apache.org/bindownload.cgi. Finally, iff you plan to compile the java code to windows dll’s for better performance (especially on production hosts), you may need to get Microsoft Visual C++ 2010 Express from http://www.microsoft.com/express/Downloads. Its free and will supply, beneath a lot of other stuff, what is expected, i.e. provides for a C++ compiler and dll linker.

Configuration

lucine-odi and ant are just zip-files, so i extracted them as follows:

• lucene-odi into F:\dbprism-lucene\lucene-odi
• ant into F:\dbprism-lucene\apache-ant-1.8.1

Microsoft Visual C++ 2010 Express, on the other hand, is just an install launcher (vc_web.exe) that will download the remaining core software only upon install and has a graphical wizard to select components and stuff. It requires you to bounce the box to complete the installation.

The ant installation instructions actually recommend to set a bunch of system variables and the path. lucene-odi has another requirements as well. However, instead of changing all that environment globally, i crafted a newly install-10g.bat file for my 10g winxp installation that features:

• a leading block of environment settings for ant, Microsoft Visual C++ 2010 Express and oracle as well as
• the necessary ant targets for the lucene-odi installation and pre-integrity test.

Do take care to safely ignore the 10g installation instructions with the Readme.txt file, that comes with the lucene-odi code stack. It took me quite a time of trial and error to find out, that this file must be out of date. For example, the ant targets ncomp-odi and test-domain-index, given there, do not exist within the current build.xml file of ant. The overall 10g instructions even miss any real installation target (should be install-odi-10g, see below). In other words, do only consult the online documentation at Lucene Domain Index for installation advice!

echo off

REM -----------------------------------------------------------------
REM - Set various homes as well as the server database sid, needed
REM - for some calls to loadjava, dropjava and statusnc.
REM -----------------------------------------------------------------
set ORACLE_HOME=F:\oracle\product\10.2.0\db_1
set ORACLE_SID=HUGOZGR
set JAVA_HOME=%ORACLE_HOME%\jdk
set ANT_HOME=F:\dbprism-lucene\apache-ant-1.8.1

REM -----------------------------------------------------------------
REM - This is necessary for executing the ncomp targets of ant that
REM - build windows dll's from java. vcvarsall.bat needs to be called
REM - in advance anytime because otherwise calling the compiler/linker
REM - (cl/link) from an oracle shell process will fail.
REM -----------------------------------------------------------------
set VC_HOME="C:\Programme\Microsoft Visual Studio 10.0"
call %VC_HOME%\vc\vcvarsall.bat

REM -----------------------------------------------------------------
REM - Iff there is more than one oracle home on the machine, setting
REM - the home alone will not route any call to loadjava, dropjava,
REM - statusnc and the like to the right oracle bin directory. The
REM - path for this shell will be reset down below (*1).
REM -----------------------------------------------------------------
set PATH_TMP=%PATH%
set PATH=%ORACLE_HOME%\bin;%PATH%

REM -----------------------------------------------------------------
REM - Change to the build dir ... this dir is also set as ${build.dir}
REM - with the build.xml main operation file of ant.
REM -----------------------------------------------------------------
cd /D F:\dbprism-lucene\lucene-odi

REM -----------------------------------------------------------------
REM - Upload, install and test your code into the database.
REM -----------------------------------------------------------------
%ANT_HOME%\bin\ant install-odi-10g
%ANT_HOME%\bin\ant test-odi

REM -----------------------------------------------------------------
REM - Build windows dll's from java for different targets of ant, the
REM - java 1.4 backport and the lucene sources.
REM -----------------------------------------------------------------
%ANT_HOME%\bin\ant ncomp-runtime-retrotranslator-sys-code
%ANT_HOME%\bin\ant ncomp-lucene-all

REM -----------------------------------------------------------------
REM - For Oracle 10g you can perform a post-installation step
REM - Report which classes are loaded to oracle but (not yet) translated
REM - to windows dll's using the ncomp compiler.
REM -----------------------------------------------------------------
%ANT_HOME%\bin\ant ncomp-status

REM -----------------------------------------------------------------
REM - Reset the for this shell, see (*1) above.
REM -----------------------------------------------------------------
set PATH=%PATH_TMP%

Another configuration object a lucene-odi installation requires, is a build.properties file which essentially holds the database connect parameters and some java configuration as follows:

db.str=__my-database-tns-name__
db.usr=LUCENE
db.pwd=LUCENE
dba.usr=sys
dba.pwd=sys
javac.debug=true
javac.source=1.4
javac.target=1.4

build.properties is expected to reside in the users home directory. iff you do not want to follow this concept for later server side application, like me, do tweak both the build.xml files in the _root_ and in the db directories of the lucene-odi code tree.

Last but not least, to have the oracle ncomp compiler, to compile the java code to windows dll’s, find the C++ compiler and linker, do adapt the visual.c.home variable with the ORACLE_HOME\javavm\jahome\Settings_os.properties file like this (the old setting has just been commented out):

#visual.c.home = C:/Program Files/Microsoft Visual Studio .NET 2003/vc7
visual.c.home = C:/Programme/Microsoft Visual Studio 10.0/VC

Installation

The configuration being done i started off install-10g.bat … here is what i learned and needed to adapt and fix:

• It was in fact nessecary here to temporarily reset the PATH environment variable in install-10g.bat because i have more than one oracle-home on that machine and the installation code does not honour the setting of ORACLE_HOME for calling loadjava (albeit, it does so for sqlplus). Iff you get some error messages like this, remember me:
  

  load-extra-libs:
       [java] arguments: '-nodefiner' '-v' '-r' '-s' '-g' 'public' '-u' 'LUCENE/LUCENE@__my-database-tns-name__' 'lib/junit-3.8.2.jar'
       [java] Error while connecting with oci8 driver to __my-database-tns-name__: oracle.aurora.server.tools.loadjava.ToolsException: The JDBC OCI8 Driver is not installed properly, use -thin to try the JDBC Thin driver: F:\oracle\product\10.2.0\db_1\BIN\ocijdbc10.dll: Die angegebene Prozedur wurde nicht gefunden
       [java] exiting  : could not open connection
  

• Proceeding, you will eventually find this, which is a typical chicken or the egg causality, that causes the ant build process to abort. Just do the
  

  create role LUCENEUSER;
  

  
  like a forward declaration from another (sysdba) session and try again. This is the original error message:
  

  F:\dbprism-lucene\lucene-odi>install-10g
  ...
    [exec] drop role LUCENEUSER
    [exec]           *
    [exec] FEHLER in Zeile 1:
    [exec] ORA-01919: Rolle 'LUCENEUSER' nicht vorhanden
  ...
  

• Another stumbling block, but much harder to w/a, was a missing java class in the test unit code (test-odi target). Since this is a 10g installation, MO does a backport of the lucene 3.x code to java 1.4 where he repacks jars and classes into new jar files. To have it in a few words, target backport-code-test misses the lucene-core-${version}-test.jar file with the jarfileset tag. The error message reads like this:
  

       [java] errors   : class org/apache/lucene/index/DBTestCase
       [java]     ORA-29521: Referenzierter Name org/apache/lucene/util/English konnte nicht gefunden werden
  

• Another attempt to execute the test-odi target may fail with a missing database connection. For running the unit tests you need to edit the sqlunit.properties file in the db directory of the lucene-odi code tree and configure the proper connection settings.
• Executing the targets ncomp-runtime-retrotranslator-sys-code and ncomp-lucene-all performs well after setting up the C++ compiler and linker appropriately. The only way to assure a successful dll creation is to check F:\dbprism-lucene\lucene-odi\tmp\ncomp.log manually as well as looking up ORACLE_HOME\javavm\deploy for the ebetually generated dll files.
• Finally, there is some problem executing the target ncomp-status. It essentially calls a java class, here oracle.jaccelerator.Status, directly from java, whereas another subcalls to java, initiated by that class, do miss the sqlnet pointer such that ORACLE_SID needs to be specified with install-10g.bat. Then, also another subcall to java prints:
  

       [java] Error while determining classes contained in tmp\lucene_ncomp_odi_minimalClassListAndTCDumper.java
       [java]     Exception oracle.aurora.sqljdecl.ParseException: Encountered "/" at line 14, column 17.
       [java] Was expecting one of:
       [java]     "extends" ...
       [java]     "implements" ...
       [java]     "{" ...
  

  
  etc. In short, i rewrote the statusnc macro with the ncomp.xml ant file to actually use the statusnc.bat command as being available with oracle bin (statusnc.bat requires absolute paths and windows path separators, hope this is ok for ant).

Conclusion

• lucene-odi installation executes on a well-layout ant target base.
• Unfortunately, the installation instructions are not up-to-date in any available resource.
• MO even supports a backport of lucene 3.x to Java 1.4, great!
• Overall, some configuration quirks and installation porting issues, nothing serios – does work.

more on application and performance tests to follow.

regards