The streaming of data block by block from your web service can have significant advantages in comparision to collecting all the data on the server side and then sending it to the client.

  1. less memory consumption
  2. faster processing time

Less Memory Consumption

In most cases when it comes to web services the data will be collected on the server and then sent to the client. And this is totally ok and works pretty well as in most cases the data transfered from and to web services is not so big.

But when it comes to transfering several hundreds of MB in one request it may not be so efficient and may even kill the server because of consuming all the available memory … ever seen the OutOfMemoryException? ;-)

// TODO OOME Exception oder Error?

But what if you just process one entity at a time and send that chunk of data to the client? Memory consumption wouldn’t be any problem at all.

Having a solution for the server is great. But you also have to take care of the client side (if it is in your hands) so that you don’t just push the memory consumption problem from the server to the client. You need to process the data on the client also block by block.

Faster Processing Time

When streaming the data from the server to the client without having to first collect the whole data on the server side the client gets the first block of data much faster and thus can process the data earlier. Thus the whole processing time is much shorter.

Jakarta JAX RS Streaming Support

JAX RS supports the streaming of data via the interface jakarta.ws.rs.core.StreamingOutput which can be returned as the response entity.

1
return Response.ok(myStreamingOutput).build();

Jdbi Streaming Support

Jdbi supports the streaming of data directly from an SQL query out-of-the-box in the Core API and also in the SQL Objects module with the interface org.jdbi.v3.core.result.ResultIterable. This interface can be used instead as a return type just like java.util.List.

The interface ResultIterable comes with multiple methods to work with the data. Just keep in mind that almost all these methods work similar as in java.util.stream.Stream as they are almost all terminal operations. With executing these terminal operations you will load and process all data from your query and cannot process the data again with the same ResultIterable object.

From the Jdbi docs:

Almost all operations on the ResultIterable interface are terminal operations. When they finish, they close and release all resources, especially the ResultSet and PreparedStatement objects.

JAX RS + Jdbi

Both parts (JAX RS and Jdbi) support the streaming of data. Now we just have to bring those two together. We need to implement StreamingOutput for this and have to have access to the stream and the Jdbi Handle object.

1
implementation here

DAO

The DAO interface (Jdbi SQL Objects) just returns a ResultIterable instead of a list.

1
ResultIterable<Data> iterate();

You can use the method stream() to get a java.util.stream.Stream object from the ResultIterable. After processing the data from the stream all the allocated resources should be freed but the Handle instance is still open. You need to pass the handle to the StreamingOutput instance so that you can call close() on the Handle instance after the processing. One way of returning both the stream and the handle is by using the Pair interface of the Apache Commons Lang project.

1
Pair.of(resultIterable.stream(), handle);

Web Service Resource

In der eigenen Resource Klasse wird die JdbiStreamingOutput Instanz als Response Entity zurückgegeben.

1
2
3
4
public Response stream() {
Pair<Stream<Customer>, Handle> result = repo.streamWithHandle();
return Response.ok(new JdbiStreamingOutput<Customer>(result, mapper)).build();
}

Eine Jackson ObjectMapper Instanz wird für die Serialisierung der Daten übergeben.

Wrap Up

@UseRowReducer cannot be used as this collects all the data on the server before streaming it and thus …

Happy streaming!

Mihael