Installing DBPrism 3.0.2.1.0 (in-Oracle-Lucene) on a WinXP-10gR2


Introduction

This post talks, shares experiences, about installing Marcelo Ochoa’s (MO, Marcelo Ochoa’s personal blog) in-oracle-lucene implementation into an oracle 10.2.0.3 on winxp. MO did a really great job in porting a raw lucene to an oracle database. His essential trick is actually to represent the file system layer, lucene usually lives in, within a blob based storage as well as employing the oracle odci-interface. Any further information is available in the online documentation here Lucene Domain Index.

Downloads

The latest version of the code stack is available as lucene-odi-bin-3.0.2.1.0.zip from http://sourceforge.net/projects/dbprism/files/odi/3.0.2.1.0/. Do not care about the sourceforge project being called DBPrism or this other code stack around being tagged ojvm, this is all history. The latest version to comprise the lucene 3.0.2 core base is lucene-odi, lucene-ojvm currently features the lucene 2.9.2 core base as just a maintainance release.

Another necessary download is ant, as a build and installation runtime environment. I sourced apache-ant-1.8.1-bin.zip from http://ant.apache.org/bindownload.cgi. Finally, iff you plan to compile the java code to windows dll’s for better performance (especially on production hosts), you may need to get Microsoft Visual C++ 2010 Express from http://www.microsoft.com/express/Downloads. Its free and will supply, beneath a lot of other stuff, what is expected, i.e. provides for a C++ compiler and dll linker.

Configuration

lucine-odi and ant are just zip-files, so i extracted them as follows:

  • lucene-odi into F:\dbprism-lucene\lucene-odi
  • ant into F:\dbprism-lucene\apache-ant-1.8.1

Microsoft Visual C++ 2010 Express, on the other hand, is just an install launcher (vc_web.exe) that will download the remaining core software only upon install and has a graphical wizard to select components and stuff. It requires you to bounce the box to complete the installation.

The ant installation instructions actually recommend to set a bunch of system variables and the path. lucene-odi has another requirements as well. However, instead of changing all that environment globally, i crafted a newly install-10g.bat file for my 10g winxp installation that features:

  • a leading block of environment settings for ant, Microsoft Visual C++ 2010 Express and oracle as well as
  • the necessary ant targets for the lucene-odi installation and pre-integrity test.

Do take care to safely ignore the 10g installation instructions with the Readme.txt file, that comes with the lucene-odi code stack. It took me quite a time of trial and error to find out, that this file must be out of date. For example, the ant targets ncomp-odi and test-domain-index, given there, do not exist within the current build.xml file of ant. The overall 10g instructions even miss any real installation target (should be install-odi-10g, see below). In other words, do only consult the online documentation at Lucene Domain Index for installation advice!

echo off

REM -----------------------------------------------------------------
REM - Set various homes as well as the server database sid, needed
REM - for some calls to loadjava, dropjava and statusnc.
REM -----------------------------------------------------------------
set ORACLE_HOME=F:\oracle\product\10.2.0\db_1
set ORACLE_SID=HUGOZGR
set JAVA_HOME=%ORACLE_HOME%\jdk
set ANT_HOME=F:\dbprism-lucene\apache-ant-1.8.1

REM -----------------------------------------------------------------
REM - This is necessary for executing the ncomp targets of ant that
REM - build windows dll's from java. vcvarsall.bat needs to be called
REM - in advance anytime because otherwise calling the compiler/linker
REM - (cl/link) from an oracle shell process will fail.
REM -----------------------------------------------------------------
set VC_HOME="C:\Programme\Microsoft Visual Studio 10.0"
call %VC_HOME%\vc\vcvarsall.bat

REM -----------------------------------------------------------------
REM - Iff there is more than one oracle home on the machine, setting
REM - the home alone will not route any call to loadjava, dropjava,
REM - statusnc and the like to the right oracle bin directory. The
REM - path for this shell will be reset down below (*1).
REM -----------------------------------------------------------------
set PATH_TMP=%PATH%
set PATH=%ORACLE_HOME%\bin;%PATH%

REM -----------------------------------------------------------------
REM - Change to the build dir ... this dir is also set as ${build.dir}
REM - with the build.xml main operation file of ant.
REM -----------------------------------------------------------------
cd /D F:\dbprism-lucene\lucene-odi

REM -----------------------------------------------------------------
REM - Upload, install and test your code into the database.
REM -----------------------------------------------------------------
%ANT_HOME%\bin\ant install-odi-10g
%ANT_HOME%\bin\ant test-odi

REM -----------------------------------------------------------------
REM - Build windows dll's from java for different targets of ant, the
REM - java 1.4 backport and the lucene sources.
REM -----------------------------------------------------------------
%ANT_HOME%\bin\ant ncomp-runtime-retrotranslator-sys-code
%ANT_HOME%\bin\ant ncomp-lucene-all

REM -----------------------------------------------------------------
REM - For Oracle 10g you can perform a post-installation step
REM - Report which classes are loaded to oracle but (not yet) translated
REM - to windows dll's using the ncomp compiler.
REM -----------------------------------------------------------------
%ANT_HOME%\bin\ant ncomp-status

REM -----------------------------------------------------------------
REM - Reset the for this shell, see (*1) above.
REM -----------------------------------------------------------------
set PATH=%PATH_TMP%

Another configuration object a lucene-odi installation requires, is a build.properties file which essentially holds the database connect parameters and some java configuration as follows:

db.str=__my-database-tns-name__
db.usr=LUCENE
db.pwd=LUCENE
dba.usr=sys
dba.pwd=sys
javac.debug=true
javac.source=1.4
javac.target=1.4

build.properties is expected to reside in the users home directory. iff you do not want to follow this concept for later server side application, like me, do tweak both the build.xml files in the _root_ and in the db directories of the lucene-odi code tree.

Last but not least, to have the oracle ncomp compiler, to compile the java code to windows dll’s, find the C++ compiler and linker, do adapt the visual.c.home variable with the ORACLE_HOME\javavm\jahome\Settings_os.properties file like this (the old setting has just been commented out):

#visual.c.home = C:/Program Files/Microsoft Visual Studio .NET 2003/vc7
visual.c.home = C:/Programme/Microsoft Visual Studio 10.0/VC

Installation

The configuration being done i started off install-10g.bat … here is what i learned and needed to adapt and fix:

  • It was in fact nessecary here to temporarily reset the PATH environment variable in install-10g.bat because i have more than one oracle-home on that machine and the installation code does not honour the setting of ORACLE_HOME for calling loadjava (albeit, it does so for sqlplus). Iff you get some error messages like this, remember me:
    load-extra-libs:
         [java] arguments: '-nodefiner' '-v' '-r' '-s' '-g' 'public' '-u' 'LUCENE/LUCENE@__my-database-tns-name__' 'lib/junit-3.8.2.jar'
         [java] Error while connecting with oci8 driver to __my-database-tns-name__: oracle.aurora.server.tools.loadjava.ToolsException: The JDBC OCI8 Driver is not installed properly, use -thin to try the JDBC Thin driver: F:\oracle\product\10.2.0\db_1\BIN\ocijdbc10.dll: Die angegebene Prozedur wurde nicht gefunden
         [java] exiting  : could not open connection
    
  • Proceeding, you will eventually find this, which is a typical chicken or the egg causality, that causes the ant build process to abort. Just do the
    create role LUCENEUSER;
    

    like a forward declaration from another (sysdba) session and try again. This is the original error message:

    F:\dbprism-lucene\lucene-odi>install-10g
    ...
      [exec] drop role LUCENEUSER
      [exec]           *
      [exec] FEHLER in Zeile 1:
      [exec] ORA-01919: Rolle 'LUCENEUSER' nicht vorhanden
    ...
    
  • Another stumbling block, but much harder to w/a, was a missing java class in the test unit code (test-odi target). Since this is a 10g installation, MO does a backport of the lucene 3.x code to java 1.4 where he repacks jars and classes into new jar files. To have it in a few words, target backport-code-test misses the lucene-core-${version}-test.jar file with the jarfileset tag. The error message reads like this:
         [java] errors   : class org/apache/lucene/index/DBTestCase
         [java]     ORA-29521: Referenzierter Name org/apache/lucene/util/English konnte nicht gefunden werden
    
  • Another attempt to execute the test-odi target may fail with a missing database connection. For running the unit tests you need to edit the sqlunit.properties file in the db directory of the lucene-odi code tree and configure the proper connection settings.
  • Executing the targets ncomp-runtime-retrotranslator-sys-code and ncomp-lucene-all performs well after setting up the C++ compiler and linker appropriately. The only way to assure a successful dll creation is to check F:\dbprism-lucene\lucene-odi\tmp\ncomp.log manually as well as looking up ORACLE_HOME\javavm\deploy for the ebetually generated dll files.
  • Finally, there is some problem executing the target ncomp-status. It essentially calls a java class, here oracle.jaccelerator.Status, directly from java, whereas another subcalls to java, initiated by that class, do miss the sqlnet pointer such that ORACLE_SID needs to be specified with install-10g.bat. Then, also another subcall to java prints:
         [java] Error while determining classes contained in tmp\lucene_ncomp_odi_minimalClassListAndTCDumper.java
         [java]     Exception oracle.aurora.sqljdecl.ParseException: Encountered "/" at line 14, column 17.
         [java] Was expecting one of:
         [java]     "extends" ...
         [java]     "implements" ...
         [java]     "{" ...
    

    etc. In short, i rewrote the statusnc macro with the ncomp.xml ant file to actually use the statusnc.bat command as being available with oracle bin (statusnc.bat requires absolute paths and windows path separators, hope this is ok for ant).

Conclusion

  • lucene-odi installation executes on a well-layout ant target base.
  • Unfortunately, the installation instructions are not up-to-date in any available resource.
  • MO even supports a backport of lucene 3.x to Java 1.4, great!
  • Overall, some configuration quirks and installation porting issues, nothing serios – does work.

more on application and performance tests to follow.

regards

Advertisements

9 comments

  1. Thanks for this instruction!!!

    Dear Guru, Dear Brainiac!!!

    Please help me to solve problem with installing DBPrism 3.0.2.1.0 ( “lucene-odi-bin-3.0.2.1.0.zip”, <a href="http://sourceforge.net/projects/dbprism/files/odi/3.0.2.1.0&quot; title "Lucene-Oracle (ODI)" ).

    OS: Windows XP SP3 x86
    DBMS: Oracle 10g (10.2.0.1.0)
    Apache ANT 1.8.2
    Microsoft Visual C++ 2010 Express
    ORACLE JAVA VERSION: 1.4.2_04

    FILE "build.properties" is configured.
    and configured environment (JAVA_HOME, ORACLE_HOME, ANT_HOME etc.)

    After launching installation Lucene into Oracle the script had been worked about 4 minutes and terminated with failure.
    After 4 minutes of execution more than 12,000 lines of code Sevaral error messages appeared.
    They are ORA-xxxxx:
    (ORA-29521: referenced name java/lang/StringBuilder could not be found
    and ORA-29534: referenced object could not be resolved).
    But, in all loaded classes all was well

    (for example:

    [java] Classes Loaded: 42
    [java] Resources Loaded: 1
    [java] Sources Loaded: 0
    [java] Published Interfaces: 0
    [java] Classes generated: 0
    [java] Classes skipped: 2
    [java] Synonyms Created: 45
    [java] Errors: 0
    )

    , except:

    [java] resolving: class org/apache/lucene/index/AllTests
    [java] errors : class org/apache/lucene/index/AllTests
    [java] ORA-29534: referenced object could not be resolved LUCENE.org/apache/lucene/index/TestDBIndex
    [java] synonym : org/apache/lucene/index/AllTests
    [java] granting : execute on class org/apache/lucene/index/DBTestCase to public
    [java] errors : class org/apache/lucene/index/DBTestCase
    [java] ORA-29521: не удалось найти ссылочное имя org/apache/lucene/util/English
    [java] synonym : org/apache/lucene/index/DBTestCase
    [java] granting : execute on class org/apache/lucene/index/TestDBIndex to public
    [java] errors : class org/apache/lucene/index/TestDBIndex
    [java] ORA-29534: referenced object could not be resolved LUCENE.org/apache/lucene/index/DBTestCase
    [java] synonym : org/apache/lucene/index/TestDBIndex
    [java] granting : execute on class org/apache/lucene/index/TestDBIndexAddDoc to public
    [java] errors : class org/apache/lucene/index/TestDBIndexAddDoc
    [java] ORA-29534: referenced object could not be resolved LUCENE.org/apache/lucene/index/DBTestCase
    [java] synonym : org/apache/lucene/index/TestDBIndexAddDoc
    [java] granting : execute on class org/apache/lucene/index/TestDBIndexDelDoc to public
    [java] errors : class org/apache/lucene/index/TestDBIndexDelDoc
    [java] ORA-29534: referenced object could not be resolved LUCENE.org/apache/lucene/index/DBTestCase
    [java] synonym : org/apache/lucene/index/TestDBIndexDelDoc
    [java] granting : execute on class org/apache/lucene/index/TestDBIndexParallel$1 to public
    [java] errors : class org/apache/lucene/index/TestDBIndexParallel$1
    [java] ORA-29534: referenced object could not be resolved LUCENE.org/apache/lucene/index/TestDBIndexParallel
    [java] synonym : org/apache/lucene/index/TestDBIndexParallel$1
    [java] granting : execute on class org/apache/lucene/index/TestDBIndexParallel$2 to public
    [java] errors : class org/apache/lucene/index/TestDBIndexParallel$2
    [java] ORA-29534: referenced object could not be resolved LUCENE.org/apache/lucene/index/TestDBIndexParallel
    [java] synonym : org/apache/lucene/index/TestDBIndexParallel$2
    [java] granting : execute on class org/apache/lucene/index/TestDBIndexParallel$3 to public
    [java] errors : class org/apache/lucene/index/TestDBIndexParallel$3
    [java] ORA-29534: referenced object could not be resolved LUCENE.org/apache/lucene/index/TestDBIndexParallel
    [java] synonym : org/apache/lucene/index/TestDBIndexParallel$3
    [java] granting : execute on class org/apache/lucene/index/TestDBIndexParallel$4 to public
    [java] errors : class org/apache/lucene/index/TestDBIndexParallel$4
    [java] ORA-29534: referenced object could not be resolved LUCENE.org/apache/lucene/index/TestDBIndexParallel
    [java] synonym : org/apache/lucene/index/TestDBIndexParallel$4
    [java] granting : execute on class org/apache/lucene/index/TestDBIndexParallel$5 to public
    [java] errors : class org/apache/lucene/index/TestDBIndexParallel$5
    [java] ORA-29534: referenced object could not be resolved LUCENE.org/apache/lucene/index/TestDBIndexParallel
    [java] synonym : org/apache/lucene/index/TestDBIndexParallel$5
    [java] granting : execute on class org/apache/lucene/index/TestDBIndexParallel to public
    [java] errors : class org/apache/lucene/index/TestDBIndexParallel
    [java] ORA-29534: referenced object could not be resolved LUCENE.org/apache/lucene/index/DBTestCase
    [java] synonym : org/apache/lucene/index/TestDBIndexParallel
    [java] granting : execute on class org/apache/lucene/index/TestDBIndexSearchDoc to public
    [java] errors : class org/apache/lucene/index/TestDBIndexSearchDoc
    [java] ORA-29534: referenced object could not be resolved LUCENE.org/apache/lucene/index/DBTestCase

    AND after that
    IN ORACLE THERE WAS FOUNDED SEVERAL JAVA INVALID OBJECTS:

    INVALID OBJECTS in scheme LUCENE:
    /86373e88_AllTests
    /dcc56b9d_DBTestCase
    /9741ab83_TestDBIndex
    /27f2c777_TestDBIndexAddDoc
    /14b7f85f_TestDBIndexDelDoc
    /71f71292_TestDBIndexSearchDoc
    /8e92669b_TestDBIndexParallel2
    /7ce6952a_TestDBIndexParallel3
    /3d2e3536_TestDBIndexParallel4
    /77f2ec61_TestDBIndexParallel5
    /d1f15c3d_TestDBIndexParallel
    /90c9812c_TestDBIndexParallel1

    INVALID OBJECTS in scheme SYS:
    mx4j/log/CommonsLogger
    mx4j/log/Log4JLogger

    LAST MESSAGE (LOG) IN ANT AFTER TERMINATED:

    [java] granting : execute on class org/tartarus/snowball/ext/Portuguese2Stemmer to public
    [java] skipping : class org/tartarus/snowball/ext/Portuguese2Stemmer
    [java] synonym : org/tartarus/snowball/ext/Portuguese2Stemmer
    [java] The following operations failed
    [java] class org/apache/lucene/index/AllTests: resolution
    [java] class org/apache/lucene/index/DBTestCase: resolution
    [java] class org/apache/lucene/index/TestDBIndex: resolution
    [java] class org/apache/lucene/index/TestDBIndexAddDoc: resolution
    [java] class org/apache/lucene/index/TestDBIndexDelDoc: resolution
    [java] class org/apache/lucene/index/TestDBIndexParallel$1: resolution
    [java] class org/apache/lucene/index/TestDBIndexParallel$2: resolution
    [java] class org/apache/lucene/index/TestDBIndexParallel$3: resolution
    [java] class org/apache/lucene/index/TestDBIndexParallel$4: resolution
    [java] class org/apache/lucene/index/TestDBIndexParallel$5: resolution
    [java] class org/apache/lucene/index/TestDBIndexParallel: resolution
    [java] class org/apache/lucene/index/TestDBIndexSearchDoc: resolution
    [java] exiting : Failures occurred during processing

    BUILD FAILED
    E:\dbprism-lucene\lucene-odi\build.xml:532: The following error occurred while executing this line:
    E:\dbprism-lucene\lucene-odi\loadjava.xml:45: Java returned: 1

    Total time: 4 minutes 13 seconds

    What was the problem in?

    Thanks for your help.

    best regards,
    Sergey S. Semka

    Like

    1. hi sergey,

      did you consider my hint like “To have it in a few words, target backport-code-test misses the lucene-core-${version}-test.jar file with the jarfileset tag” for the build.xml file? Actually, unresoved externals are mostly caused by not correctly integrating all needed libraries. The tweaked target backport-code-test looks like this:

      <target name=”backport-code-test”
      depends=”backport-code-odi”
      description=”Downgrade Lucene JDK1.5 code 1.4″>
      <delete dir=”tmp”/>
      <mkdir dir=”tmp”/>
      <retrotranslator target=”1.4″ destjar=”lucene-odi-test-${version}.jar”
      support=”ThreadLocal.remove” smart=”true” failonwarning=”false”>
      <jarfileset dir=”lib” includes=”lucene-odi-*-test.jar,lucene-core-${version}-test.jar” />
      <classpath>
      <fileset dir=”${ORACLE_HOME}/jdk/jre/lib” includes=”**/*.jar”/>
      <fileset dir=”bin”>
      <include name=”retrotranslator-runtime-1.2.9.jar” />
      <include name=”backport-util-concurrent-3.1.jar” />
      </fileset>
      <fileset dir=”.” includes=”lucene-all-${version}.jar” />
      <fileset dir=”.” includes=”lucene-odi-all-${version}.jar” />
      <fileset dir=”lib” includes=”junit-3.8.2.jar” />
      <fileset dir=”lib” includes=”commons-codec-1.4.jar” />
      <fileset dir=”${ORACLE_HOME}/rdbms/jlib” includes=”CartridgeServices.jar” />
      <fileset dir=”${ORACLE_HOME}/rdbms/jlib” includes=”ODCI.jar” />
      <fileset dir=”${ORACLE_HOME}/rdbms/jlib” includes=”xdb.jar” />
      <fileset dir=”${ORACLE_HOME}/jdbc/lib” includes=”ojdbc14.jar” />
      <fileset dir=”${ORACLE_HOME}/lib” includes=”xmlparserv2.jar” />
      <fileset dir=”${ORACLE_HOME}/javavm/lib” includes=”aurora.zip” />
      </classpath>
      </retrotranslator>
      </target>

      regards peter

      Like

  2. Thank you very much for your proposed solution.

    I added you advise the following code: lucene-core-${version}-test.jar in “…\lucene-odi\build.xml”
    and my problem was solved.

    <jarfileset dir=”lib” includes=”lucene-odi-*-test.jar,lucene-core-${version}-test.jar” />

    —————————

    Please answer: with the original file BUIL.XML using “ANT” everything should normally be installed?
    Or required for installing “Oracle Lucene Domain Index” in Oracle 10g changes (configure) to the file Build. xml as suggested?

    Like

  3. Ooops…

    I find two INVALID OBJECTS in scheme SYS:
    mx4j/log/CommonsLogger
    mx4j/log/Log4JLogger

    These two objects are not compiled….

    This is a big problem?

    What was the problem in?

    Other invalid object does not exist.
    In the scheme LUCENE all objects is VALID ”

    Thanks for your help.

    Like

  4. hi!

    these two classes are marked invalid with my installation too (although i did’nt notice so far). obviously, the mx4j package misses some externals from log4j as follows (in german but you can stick to the error codes):


    alter java class "mx4j/log/CommonsLogger" resolve;
    show errors
    Fehler bei JAVA CLASS "mx4j/log/CommonsLogger":
    LINE/COL|ERROR
    --------|-----------------------------------------------------------------
    0/0 |ORA-29521: Referenzierter Name org/apache/commons/logging/Log konnte nicht gefunden werden
    0/0 |ORA-29521: Referenzierter Name org/apache/commons/logging/LogFactory konnte nicht gefunden werden


    alter java class "mx4j/log/Log4JLogger" resolve;
    show errors
    Fehler bei JAVA CLASS "mx4j/log/Log4JLogger":
    LINE/COL|ERROR
    --------|-----------------------------------------------------------------
    0/0 |ORA-29521: Referenzierter Name org/apache/log4j/Logger konnte nicht gefunden werden
    0/0 |ORA-29521: Referenzierter Name org/apache/log4j/Priority konnte nicht gefunden werden
    0/0 |ORA-29521: Referenzierter Name org/apache/log4j/Level konnte nicht gefunden werden

    i did not set up lucine-odi installation, just tweaked it a bit so i can’t tell you iff this is a problem at all. me, i did’nt experience any logging problems so far.

    regards

    Like

    1. güten Tag! Hi!

      Thank you for your reply!

      I think the problem is that the file “…\bin\mx4j.jar/mx4j\log\CommonsLogger.class” contains any references to Apache library (org/apache/commons/logging/Log). But this library not loaded in Oracle. (Or incorrectly loaded in DB). Yet the conversation started. And I think someone will help us…

      ———————

      Please answer, you use Lutsene for what purpose? (full text search, information system, a web site; search engine, text data files). How much data in your database? (How many records in the database?).

      By the way, what country are you from? (Marcelo Ochoa from Tandil, Buenos Aires, Argentina; I’m from Saint Petersburg, Russia).

      best regards,
      Sergey

      Like

  5. Hi

    First of all thanks a ton for this article. This definitely resolved my most of the installation issues. however for “ant install-odi-10g” command i got multiple errors as mentioned below.

    [java] Classes Loaded: 0
    [java] Resources Loaded: 1
    [java] Sources Loaded: 0
    [java] Published Interfaces: 0
    [java] Classes generated: 0
    [java] Classes skipped: 385
    [java] Synonyms Created: 386
    [java] Errors: 0

    It seems that some classes have been skipped. Can you please help here.

    Regards

    Like

    1. hi lazy dba,
      the ldi guys do not seem to worry much about uploading all the lucene code stack into the database such that you may have a couple / a lot of unresolved or even stale class references. In fact, i did raise the same question during my tests … like: iff there is no dangling reference, there is no upload – compilation – potential execution – etc. problem. Maybe i’m pedantic, but so far, checking the loadjava output is not a reliable success factor.
      Well, this is one of the reasons i meanwhile refrained from continuing any ldi related work. Just take a look at the input parameter checking as currently implemented, it just does not exist. Ldi is a true prove that lucene will performantly run in an aurora vm but the way the code has been implemented is not my way of doing these things.
      regards

      Like

      1. Thanks for Quick reply Bitbach.

        Even I did ran some test related to lucene & it worked. But Not really sure if anything is going to fail or not. I am planning to execute some more test cases and see if everything is as expected or not.

        Regards

        Like

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s