Home

Awesome

Web Archive Discovery

These are the components we use to data-mine and index our ARC and WARC files and make the contents explorable and discoverable.

Java CI Maven Build

Maven Central

Documentation

See the wiki.

Running the development Opensearch Server

The Opensearch part is also usable for Elasticsearch 7.10.2 and may usable for older versions (with minor modifications). You can start it with the provided docker-compose file. After checkout do the following steps in a shell

$ cd warc-indexer/src/main/opensearch/os1
$ docker-compose up -d

Initalize the index

To use the cluster you need to create an index. You can do it by calling

$ curl --insecure --user admin:admin -H 'Content-Type: application/json' -XPUT https://localhost:9200/warcdiscovery/  -d @schema.json

this call creates the index with the schema.json which you can use with warcindexer. You can delete the index by calling

$ curl --insecure --user admin:admin -XDELETE https://localhost:9200/warcdiscovery

Solr-schema ported to Opensearch

The Solr-schema was as close as possible ported to Opensearch. There are just a few small differences:

Indexing a WARC file

Use the following line if you want to populate the opensearch index:

$ java -jar target/warc-indexer-*-jar-with-dependencies.jar -e https://localhost:9200/warcdiscovery/ --user admin --password admin src/test/resources/wikipedia-mona-lisa/flashfrozen-jwat-recompressed.warc.gz

License

Overall, GNU General Public License Version 2, but some sub-components are Apache Software License, Version 2.0.