Awesome
LDES Server
The Linked Data Event Stream (LDES) server is a configurable component that can be used to ingest, store, transform and (re-)publish an LDES. The LDES server was built in the context of the VSDS project in order to easily exchange open data.
Table Of Contents
Structure of the application
LDES Server Dependency Graph With All Profiles Activated: Above is a dependency graph of the LDES server with all the profiles activated. The graph is color-coded as follows:
- Orange: Main functionalities
- Blue: Interfaces
- Lavender: Plugin fragmentations
The ldes-server-domain
is the core domain module of the LDES server and is always loaded in.
To keep the graph clean, it is not shown.
Next to that, the postgres-liquibase
module will always be loaded in to assure correct database creation.
The LDES server is built using the Spring Boot framework and is structured as a multi-module Maven project. Each maven profile represents a different functionality of the LDES server that can be toggled. The default exported image contains all the profiles, but a custom image can be created with only the needed dependencies.
Profile | Dependencies | Description |
---|---|---|
fragmentation | postgres-pagination-repository | Allows basic fragmentation (pagination) |
maintenance | postgres-maintenance-repository | Allows the LDES server to perform maintenance operations on its Event Streams and Members: compaction, retention, deletion. |
Interfaces | ||
http-admin | ldes-server-admin-rest ,postgres-admin-repository * | Gives access to REST API to create and manage Event Streams and Views. |
http-ingest | ldes-server-ingest-rest ,postgres-ingest-repository | Gives access to REST API to ingest members into the LDES. |
http-fetch | ldes-server-fetch-rest ,postgres-fetch-repository | Gives access to REST API to fetch Event Streams, its Views and pages. |
Plugin Fragmentations | ||
fragmentation-timebased | ldes-server-fragmentation-timebased-hierarchical | Allows fragmentation in based on a timebased property. |
fragmentation-geospatial | ldes-server-fragmentation-geospatial | Allows fragmentation in based on a geospatial property. |
fragmentation-reference | ldes-server-fragmentation-reference | Allows fragmentation in based on a textual property. |
*: The postgres-admin-repository
, as shown by the dependency graph, will be loaded in by the other above-mentioned functionality profiles.
But when used separately, it needs to be loaded in manually.
How To Run
To run the LDES server, we refer to the versioned documentation available here.
Maven
To locally run the LDES server in Maven, move to the ldes-server-application
directory and run the Spring Boot
application.
This can be done as follows:
NOTE: Due to an authorisation issue in GitHub packages, the easiest way is to first build the project using the following command:
mvn install
cd ldes-server-application
mvn spring-boot:run -P{profiles (comma separated with no spaces) }
The needed profiles can be found in the Structure of the application section.
Developer Information
How To Build
For compilation of the source code, execute the following command
mvn clean compile
How To Test and View Coverage
Below the three options to run tests (only unit test, only integration tests and both unit and integration tests) are
listed.
To view the coverage of each option it's sufficient to add the option -Pcoverage
after the command.
This generates a file in the target
folder called jacoco.exec
. This file reports the code coverage.
The combined coverage can also be seen via SonarCloud.
Unit and Integration Tests
For running all the tests of the project, execute the following command
mvn clean verify
Only Unit Tests
For running the unit tests of the project, execute the following command
mvn clean verify -Dintegrationtestskip=true
Only Integration Tests
For running the integration tests of the project, execute the following command
mvn clean verify -Dunittestskip=true
Tracing and Metrics
Additionally, it is possible to keep track of metrics and tracings of the LDES Server. This will be done through a Zipkin exporter for traces and a Prometheus endpoint for Metrics.
The exposed metrics can be found at /actuator/prometheus
.
Both traces and metrics are based on OpenTelemetry standard
To achieve this, the following properties are expected
Local Tracing and Metrics
management:
tracing:
sampling:
probability: 1.0
zipkin:
tracing:
endpoint: "zipkin endpoint of collector"
endpoints:
web:
exposure:
include:
- prometheus
The export of traces can be disabled with the following parameter:
management:
tracing:
enabled: false
Pyroscope instrumentation
To enable pyroscope, add the following to the application.yml file:
pyroscope:
agent:
enabled: true
Note that this does not work when running the server locally on a Windows device.
The normal pyroscope properties can be found here These properties should be added to the env variables.
Using Docker
SPRING_SLEUTH_OTEL_EXPORTER_JAEGER_ENDPOINT="endpoint of collector"
MANAGEMENT_ENDPOINTS_WEB_EXPOSURE_INCLUDE="prometheus"
MANAGEMENT_TRACING_SAMPLING_PROBABILITY="1.0"
MANAGEMENT_ZIPKIN_TRACING_ENDPOINT="zipkin endpoint of collector"
The export of traces can be disabled with the following parameter:
MANAGEMENT_TRACING_ENABLED=false
Health and Info
To allow more visibility for the application, it is possible to enable a health and info endpoint.
This health endpoint provides a basic JSON output that can be found at /actuator/health
that provides a summary of the
status of all needed services.
An additional info endpoint is also available which shows which version of the application running and its deployment date.
Local Health and Info
The following config allows you to enable both the info and health endpoints.
management:
endpoints:
web:
exposure:
include:
- health
- info
health:
defaults:
enabled: false
mongo:
enabled: true
dcat:
enabled: true
endpoint:
health:
show-details: always
Docker
MANAGEMENT_ENDPOINTS_WEB_EXPOSURE_INCLUDE="health, info"
MANAGEMENT_HEALTH_DEFAULTS_ENABLED=false
MANAGEMENT_HEALTH_MONGO_ENABLED=true
MANAGEMENT_HEALTH_DCAT_ENABLED=true
MANAGEMENT_ENDPOINT_HEALTH_SHOW-DETAILS="always"
Exposing of details
With the above config, where management.endpoint.health.show-details=true
, all the details of the declared health
components are exposed. The details that should not be exposed, can be hidden by two ways.
-
Disabling unnecessary details
The details that should not be exposed, can be disabled. This can be achieved by disabling all the defaults and enabling the required details only, just like the above config, or by enabling the required details only. This can be done with the following property:management.endpoints.<component-name>.enabled=<true/false>
-
Declaring a group and include there all the desired details
With the following config, a group is created that exposes all its details of the components within this group. This ensures that other details that are not part of this group are not exposed.management: endpoint: health: show-details: always group: dcat-validity: show-details: always include: dcat
Logging
The logging of this server is split over the different logging levels according to the following guidelines.
- TRACE: NONE
- DEBUG: Standard operations like: create fragment, ingest member, assign member to fragment
- INFO: NONE
- WARN: Potentially unintended operations like: Duplicate Member Ingest, ...
- ERROR: All Exceptions
Logging configuration
The following config allows you to output logging to the console. The trace id and span id can be included in the logging, if enabled via the tracing config. Further customization of the logging settings can be done using the logback properties. A use case for this can be sending the logs to Loki for example.
logging:
pattern:
console: "%5p [${spring.application.name:LDESServer4J},%X{traceId:-},%X{spanId:-}]"
level:
root: INFO
The following config enables and exposes the loggers endpoint.
management:
endpoint:
loggers:
enabled: true
endpoints:
web:
exposure:
include:
- loggers
To change the logging level of the application at runtime, you can send the following POST request to the loggers endpoint. Replace [LOGGING LEVEL] with the desired logging level from among: TRACE, DEBUG, INFO, WARN, ERROR.
curl -i -X POST -H 'Content-Type: application/json' -d '{"configuredLevel": "[LOGGING LEVEL]"}'
http://localhost:8080/actuator/loggers/ROOT