Developer Guide Resource Query Language (RQL)

RQL Parser

This Maven module contains the antlr4 grammar and the Bosch Semantic Stack RQL parser to create the RqlQueryModel from a string or vice versa. For more information on antlr4, refer to https://github.com/antlr/grammars-v4.

The parser is the central tool that uses antlr4 code to analyze RQL queries and convert them into an RqlQueryModel. Therefore, this module enables the interpretation of RQL queries in string format and transforms them into an internal model (RqlQueryModel) that can be further processed.

To include the plugin, use the following dependency:

<dependency>
    <groupId>com.boschsemanticstack</groupId>
    <artifactId>semanticstack-rql-parser</artifactId>
    <version>{version}</version>
</dependency>

Creation of a model

To create a model we can use RqlParser.builder(). This maps the structure such as select, filter and option.

final RqlQueryModel model = RqlParser.builder()
      .select( "att1", "att2", "att3.subAtt4" )
      .filter( RqlBuilder.and(
            RqlBuilder.eq( "att2", "theSame" ),
            RqlBuilder.or(
                  RqlBuilder.lt( "att1", 5238907523475022349L ),
                  RqlBuilder.not(
                        RqlBuilder.gt( "att1", new BigInteger( "12345678901234567890123456789012345678901234567890" ) )
                  )
            ) ) )
      .sort( RqlBuilder.asc( "att1" ), RqlBuilder.desc( "att2" ) )
      .limit( 0, 500 )
      .build();

Creating a string representation of the model

To create a string representation of the model, we can use the toString() method of the RqlParser.

final String representation = RqlParser.toString( model );

Single argument or body

Do you transport the query via HTTP body or as a single query attribute? To parse such a query from REST:

private void someRestEndpoint( final String theWholeQuery ) {(1)

   // This highly depends on your REST backend therefore no API call to do this is provided
   final RqlQueryModel from = RqlParser.from( Optional.ofNullable( theWholeQuery ).orElse( "" ) );

   // Do something with query model
}

1	The whole query taken from the HTTP body or a single query argument.

As multiple query parameters

Parsing a query transported component-wise (as three independent query arguments for the select, filter, and option operators) requires re-assembly:

private void someRestEndpoint(
      final String selectParam, (2)
      final String filterParam, (2)
      final String optionParam ) { (2)

   final String queryString = Stream.of(
               Optional.ofNullable( selectParam ).map( s -> "select=" ), (1)
               Optional.ofNullable( filterParam ).map( s -> "filter=" ), (1)
               Optional.ofNullable( optionParam ).map( s -> "option=" )  (1)
         )
         .filter( Optional::isPresent )
         .map( Optional::get )
         .collect( Collectors.joining( "&" ) );

   final RqlQueryModel from = RqlParser.from( queryString );

   // Do something with query model
}

1	The whole query taken from the HTTP body or a single query argument.
2	The parameter names on your API do not even need to be named `select,filter or query` they could be named `fields,restrictions,settings`.

To query a RQL-supporting service:

   private Observable<RestResponse> getSomeResourceWithRqlUsingQueryParameters( final RqlQueryModel query ) {

      final Map<String, String> queryParameters = RqlParser.toQueryParameters( query );

      return someRestClient.post( "someAddress" )
            .addQueryParam( "select", queryParameters.get( "select" ) ) (1)
            .addQueryParam( "filter", queryParameters.get( "filter" ) ) (1)
            .addQueryParam( "option", queryParameters.get( "option" ) ) (1)
            .toObservableResponse();
   }

1	Be mindful, that each parameter part may be empty (not present in Map)

Adding restrictions to a model

Often you need to enforce certain restrictions on a query that may not be circumvented by the query itself. For example, restricting the query to a certain tenant or time frame. To do this the Rql companion object supports adding restrictions around the original query (as a top-level and(…)):

      final RqlQueryModel modelWithRestriction = RqlParser.addRestriction(
            model,
            RqlBuilder.eq( "att2", "fizzBuzz" ), (1)
            RqlBuilder.or( RqlBuilder.ne( "att3", 42 ) ) (2)
      );

1	If the given model already has `and` as its top restriction a new `and` clause is created using the original and new restrictions.
2	More than one added restriction is added to only one new `and` filter.

Accessing a paged service

Sometimes a service you want to access limits its pagesize to something smaller than you need.

In this case you need to query multiple times to get all data. This can be expressed elegantly using the following construct:

   public Observable<String> getSomethingRemote( final RqlQueryModel query ) { (1)
      final int pageSize = 170;  // entirely dependent on remote service
      return getSomeResourceRecursive( RqlParser.getPagedQuery( query, pageSize ) );
   }

   private Observable<String> getSomeResourceRecursive( final Iterator<RqlQueryModel> pagedQuery ) {

      if ( pagedQuery.hasNext() ) {
         final RqlQueryModel next = pagedQuery.next();
         return someRestClient.post( "someAddress" )
               .withBody( new RqlToStringWriter().visitModel( next ) )
               .toObservableResponse()
               .flatMap( response -> doRecursionIfMoreMeasurementsAvailable(
                     pagedQuery,
                     next.getOptions().getSlice().get().limit(), (2)
                     response )
               );
      }
      return Observable.empty();
   }

   private Observable<String> doRecursionIfMoreMeasurementsAvailable(
         final Iterator<RqlQueryModel> pagedQuery,
         final long pageSize,
         final RestResponse response ) {

      if ( response.getResponseCode() != 200 ) {
         return Observable.error( new RuntimeException( "Remote service responded with " + response.getResponseCode() ) );
      }

      final List<String> results = response.getBodyAsList();
      Observable<String> observableResults = Observable.fromIterable( results );

      if ( results.size() == pageSize && pagedQuery.hasNext() ) { (3)
         observableResults = observableResults.concatWith( Observable.defer((4)
               () -> getSomeResourceRecursive( pagedQuery ) )
         );
      }

      return observableResults;
   }

1	The original Query may well have defined it’s own slice (e.g. [25:701]) this is honored throughout the process.
2	Always exists in models generated by `Rql.getPagedQuery`.
3	Continue requesting as long as a) the remote service returns as many items as requested and b) there are still more items to be requested in the original query.
4	The deferral is needed so another request is only made after we get to this element. Otherwise, a `.take(15)` further up the stream would not take effect until all elements had been fetched.

RQL to QueryDSL

The rql-2-querydsl module provides a bridge from an RQL model to a Querydsl model. Querydsl strives to be a technology-agnostic, typesafe query language that can be mapped to different concrete technologies such as JPA, SQL and MongoDB. Using the bridge in combination with the binding of Querydsl for Spring Data enables easy usage of RQL to access a database through Spring Data repositories.

JAXB annotations (or similar Jackson or Gson annotations) are not yet considered during the translation.

To include the plugin, use the following dependency:

<dependency>
    <groupId>com.boschsemanticstack</groupId>
    <artifactId>semanticstack-rql-2-querydsl</artifactId>
    <version>{version}</version>
</dependency>

Features

Select / Projection

Not yet supported.

Filtering

The bridge currently supports the operators:

Comparison: eq, in, ne, gt, ge, lt, le, like and likeIgnoreCase
Logical: and, or and not

Top-level attributes, nested attributes, and collections of attributes can be referenced in query expressions.

In the following example, filters are defined on attributes of a Person entity. This Person entity has a firstName top-level attribute, an attribute addresses that references a sub-entity of type Address and a list of Hobby entities.

filter=eq(firstName,"John")
filter=eq(address.zipCode,1234)
filter=like(hobbies.description,"?iking*")

If there are multiple constraints on entries of a collection, all constraints are applied to each entry.

That means the following query matches persons who have a hobby with the name "ships" and that hobby’s description contains "?iking*":

filter=and(like(hobbies.description,"?iking*"),eq(hobbies.name,"ships"))

Thus the semantic is similar to joining the hobby once in a relational database.

Ordering

Ordering is supported for an entity’s attributes.

For example, the following query would sort persons by zipCode in descending and name in ascending order:

option=sort(-address.zipCode,+name)

Pagination / Limits

Currently, not directly supported. There is no generic way to use Querydsl for that.

However, the QueryModelToQueryDSL instance contains this information afterwards so that it can be used manually with your concrete access solution (e.g., you can use this information with Spring Data repositories). Access paging information like this:

QueryModelToQueryDSL transformedQueryModel = ... // transform query
Optional<ISlice> paging = transformedQueryModel.getPagination();

Usage & Examples

Required dependencies

         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">

   <modelVersion>4.0.0</modelVersion>
   <parent>
...
         <artifactId>semanticstack-rql-parser</artifactId>
      </dependency>
      <dependency>
         <groupId>com.h2database</groupId>
         <artifactId>h2</artifactId>
      </dependency>
      <!-- Optional -->
      <dependency>
         <groupId>org.springframework.boot</groupId>
         <artifactId>spring-boot-starter-data-jpa</artifactId>

		<!-- In case of using JPA: spring-data-jpa is required -->
      <dependency>
         <groupId>org.springframework.boot</groupId>
         <artifactId>spring-boot-starter-data-jpa</artifactId>
      </dependency>

Set up generation of Querydsl metamodel classes

In order to query object graphs in a typesafe way, Querydsl relies on (usually automatically generated) metamodel classes. The meta model classes use the naming prefix Q to distinguish them from the model classes, e.g., a class Entity would have a metamodel class QEntity.

The rql-2-querydsl bridge relies on the metamodel classes as well. See the official documentation on how to set up Maven to generate classes for use with JPA, SQL, etc. The following example shows how to use RQL via Querydsl on Spring Data MongoDB repositories.

You need to generate the meta model classes using the Spring Data MongoDB specific annotation processor; add the following plugin configuration to the pom.xml:

QueryDSL since 5.0.0

<!-- used to build QClasses during normal compile time -->
<dependency>
   <groupId>com.querydsl</groupId>
   <artifactId>querydsl-apt</artifactId>
   <version>${querydsl.version}</version>
   <classifier>jpa</classifier>
   <scope>provided</scope>
</dependency>

Calling mvn compile will then generate a class QEntity and corresponding Q-classes for the classes of subentities that are direct parts of Entity. If your entity class hierarchy is nested deeper than one level, you need to annotate subentity classes with @QueryEmbeddable for the annotation processor to generate Q-classes for them, e.g.:

import com.querydsl.core.annotations.QueryEmbeddable;

@QueryEmbeddable
public class SubEntity {
}

Generating a Querydsl Predicate from an RQL expression

Creating a Querydsl Predicate which you can then use to perform the actual query consists of two parts: parsing the raw RQL query and transforming it into a Predicate for your underlying store.
As of now the transformation process is not completely independent of the underlying store, but you’ll be guided by the API.

import com.bosch.bci.rql.model.v1.IQueryModel;
import com.bosch.bci.rql.parser.v1.RqlParser;
import querydsl.rql.com.boschsemanticstack.QueryModelToQueryDSL;
import com.querydsl.core.types.Predicate;

public void example() {
	String rqlQuery = "filter=eq(firstName,\"John\")";
	IQueryModel queryModel = RqlParser.from(rqlQuery);

	// JPA-specific
	QueryModelToQueryDSL bridge = QueryModelToQueryDSL.forJpa(QEntity.entity, queryModel);

	// Generic (e.g. MongoDB)
	QueryModelToQueryDSL bridge = QueryModelToQueryDSL.forGenericStore(QEntity.entity, queryModel);
	...
	Optional<Predicate> optionalPredicate = bridge.getPredicate();
}

The above shown methods are convenience shortcuts, also a builder can be used which allows for more fine grained configuration. See chapter Explicit type conversions for an example.

Executing an RQL query on a Spring Data MongoDB repository

To be usable with Querydsl, the repository must extend the QuerydslPredicateExcecutor interface, e.g.:

import org.springframework.data.mongodb.repository.MongoRepository;
import org.springframework.data.querydsl.QuerydslPredicateExecutor;

public interface EntityRepository extends MongoRepository<Entity,String>,
                                          QueryDslPredicateExecutor<Entity> {
}

By using standard Spring dependency injection, the repository can be made available and queried using the generated Querydsl predicate, e.g.:

EntityRepository repository = ...;
Optional<Iterable<Entity>> result = optionalPredicate.map(repository::findAll);

Use paging and sorting with Spring Data

The QueryModelToQueryDSL offers the information about the requested limit and the sort options:

	public List<OrderSpecifier<?>> getOrdering() {...}

	public Optional<ISlice> getPagination() {...}

Spring Data provides a PagingAndSortingRepository which accepts a PageRequest containing the sort and page information. In the examples there is a QueryDslRepositoryFilter which converts the OrderSpecifier and the ISlice into a PageRequest:

   public Page<T> findWithQuery( final QueryModelToQueryDSL queryDsl ) {
      if ( queryDsl == null ) {
         throw new IllegalArgumentException( "Query must not be null" );
      }
      return find( queryDsl );
   }

   private Page<T> find( final QueryModelToQueryDSL queryDsl ) {
      final PageRequest pageRequest = createPageRequest( queryDsl );
      final Optional<Predicate> predicate = queryDsl.getPredicate();
      return predicate.map( p -> repository.findAll( p, pageRequest ) ) //
            .orElse( repository.findAll( pageRequest ) );
   }

   private PageRequest createPageRequest( final QueryModelToQueryDSL queryDsl ) {
      // Somewhat inconsistent in the API: ISLice is from query.dsl, OrderSpecifier from spring data
      final Optional<RqlSlice> pagination = queryDsl.getPagination();
      final List<OrderSpecifier<?>> ordering = queryDsl.getOrdering();

      final List<Sort.Order> sortOrder = ordering.stream()
            .map( QueryDslRepositoryFilter::convert )
            .filter( Objects::nonNull )
            .collect( Collectors.toList() );

      return pagination.map( p -> PageRequest.of( (int) p.offset(), (int) p.limit(), Sort.by( sortOrder ) ) )

Explicit type conversions

It is possible to apply explicit type conversions during the translation from RQL to Querydsl. This is e.g. necessary, if your domain model uses types that can’t directly be assigned from the values parsed from the RQL query. A typical example is a UUID or a date, which both will be provided as strings.

To apply type conversions, the builder for the QueryModelToQueryDSL bridge must be used:

QueryModelToQueryDSL bridge =
    // example for JPA, works the same for generic stores
    RqlToQueryDslConverterBuilder.forJpa(QEntity.entity)
                                 .withTypeConverter(UUID.class, UUID::fromString)
                                 .build()
                                 .applyTo(queryModel);

Any number of type converters can be registered using method chaining.

Customize Paths via delegate methods

For more information see QueryDSL docs

A small example to demonstrate the feature.

public class Foo {
   private FooBar fooBar;
}

public class FooBar{
   private Bar bar;
}

public class Bar{
   private String name;
}

The main entity to handle is Foo. If you would to search for a bar name the search query/path is always fooBars.bar.name. There are a few reasons to customize this path. e.g. only bar.name.

This is possible with a QueryEntity and delegate methods.

The method name must match the path element. This means that we need the following new methods for delegation in this case.

In the class Foo, the path fooBars should be able to be specified directly with bar. For that we need a method bar.

@QueryEntity
public class FooExtension {

@QueryDelegate( QFoo.class )
public static SetPath<FooBar, QFooBar> bar( final QFoo foo ) {
   return foo.fooBars;
   }
}

In the class FooBar, the path bar.name should be able to be specified directly with name. For this we need a method name

@QueryEntity
public class FooBarExtension {

@QueryDelegate( QFooBar.class )
public static StringPath name( final QFooBar fooBar ) {
   return fooBar.bar.name;
   }
}

Now the apt plugin will go and generate new methods in the generated classes and use these static methods. The extension must be outside a package which is not re-created by the apt plugin.

Exclude Paths and Operation query type

Declared allowed paths

With the possibility of the 'QueryType' you can skip and manipulate the apt generation. For more information see QueryDSL docs

public class Foo {
   @QueryType(NONE)
   private String id;
   @QueryType(SIMPLE)
   private String simple;
   private String name;
   private Bar bar;
}

public class Bar{
   private String name;
}

In this example we can search for 'Foo#name' and 'Bar#name'. For Path 'Foo#simple' you can only use 'eq' and 'ne'. Path for 'Foo#id' is not generated

Restrict wildcard for Like expression

With the possibility of the custom annotation 'WildcardCount' and 'RqlPattern', you can add restriction for the like expression. The idea is that for some like operations the number of wildcards should be limited depending on the field. More complicated patterns can be added via pattern e.g. wildcards may only be used at the beginning.

Wildcard count

The idea is that for some like operations the number of wildcards should be limited depending on the field e.g. for this field just 2 wildcards are allowed.

Example:

public class Foo {
   @WildcardCount(count = 2)
   private String id;

   @WildcardCount(count = 1)
   private String name;

   private Bar bar;
}

public class Bar{
   private String name;
}

In this example we limit the 'Foo#id' and 'Foo#name'. For 'Foo#id'2 wildcards are allowed. For 'Foo#name' one wildcard is allowed. The 'bar#name' has not limits.

Wildcard Pattern

More complicated patterns can be added via pattern e.g. wildcards may only be used at the beginning or end.

Example:

public class Foo {
   private String id;
   private String name;

   private Bar bar;
}

public class Bar{
   @RqlPattern(regex = "^*?[^*]+$|^[^*]+?*?$")
   private String name;
}

In this example we limit the 'Bar#name'. For 'Bar#name' the wildcard is just allowed at the beginning. The 'id' and 'name has not limits.