i means integration - Sending data with STOMP client

In the last posts we have set up all necessary building blocks for sending data to a message queue on the Apache Artemis instance on IBM i. Now we are putting all those pieces together and write an ILE RPG progam which sends data to a message queue with the STOMP service program.

We are picking up the case from the first part of this blog series. We want to inform that a new auction has started including the items that will be part of the auction.

1  **FREE
2
3  ctl-opt main(main);
4
5  /include 'stomp.rpginc'
6
7  dcl-ds auction_t qualified template;
8     id uns(20);
9     title varchar(100);
10     description varchar(1000);
11     items likeds(auction_item_t) dim(10);
12     auctionStart timestamp;
13     auctionEnd timestamp;
14 end-ds;
15
16 dcl-ds auction_item_t qualified template;
17     id uns(10);
18     name varchar(100);
19     quantity uns(3);
20 end-ds;
21
22 dcl-proc main;
23     dcl-s client pointer;
24     dcl-ds auction likeds(auction_t);
25
26     auction = fillAuctionData();
27    
28     client = stomp_create('localhost' : 40002);
29     stomp_open(client);
30     stomp_command_connect(client : 'auction_user' : 'uwillneverknow');
31     stomp_command_send(client : 'auction-events' : %addr(auction) : %size(auction) : STOMP_MIME_BINARY);
32     stomp_command_disconnect(client);
33     stomp_close(client);   
34     
35     on-exit;
36         stomp_finalize(client);
37 end-proc;

This looks like a lot of code but the main procedure is just 16 lines. So it is not much at all. Let’s look at the code line by line.

Line 1 : We are writing fully free RPG code. We are not half-assing this already from the start ;-)
Line 3 : We will compile this program in two steps. Thus we only declare the main procedure in the control options.
Line 5 : The STOMP copybook needs to be included so that we can use the constants and prototypes from the STOMP project. Everything else will be added to the compile commands.
Line 7 - 20 : The definition of the auction data which will be sent to the message queue.
Line 26 : The data is loaded here. Nothing fancy.
Line 28 : An instance of the STOMP client is created. The client is accessed via the pointer. The internal data of client doesn’t matter to us. We are only using the procedures from the STOMP project anyway. The host and port we want to connect to are passed to the create procedure. The port should be the same we used in the Artemis configuration for our STOMP acceptor.
Line 29 : A socket connection to the message broker is created.
Line 30 : To connect on a (STOMP) protocol level we need to send a STOMP frame/command to the server. This also includes the credentials used to login to the message broker. For an anonymous connect we can just pass the client.
Line 31 : Now here we are sending the data at last. It will be sent as a block of bytes. The MIME type also states that it is no text data but raw bytes which are the payload of the message.
Line 32 : If no more message are to be sent the connection can be closed (on a protocol level).
Line 33 : The socket connection needs to be closed.
Line 36 : Last but not least the memory allocated by the client needs to be freed.

We can see the message in the sink queue we configured in the Artemis server. But there are some suboptimal things we can improve.

Character Encoding

The data sent to the message queue is in the CCSID of the current job. By default this is stated nowhere in the message. So the receiver of the message has no idea what the encoding of the message content is. One way of dealing with this is adding the charset parameter to the Content-Type with the correct charset for the current CCSID, like application/octet-stream; charset=x-ebcdic-germany.

But EBCDIC is not a common character encoding and thus not a good encoding for data exchange as the sender doesn’t know who will receive the message.

Sending the message in the JSON format

The message we sent in the above code is hard to read as is because it was sent as a sequence of bytes. We used the data structure directly as the format of the data sent to the message queue so every message receiver must also know the definition of the used data structure. As this is RPG specific it is not a good choice for data exchange. It would be better to use a data format which is not programming language specific … like JSON.

Currently many services in the whole world exchange data in JSON format. We will also use this as it is a very common and nice format which is also easily readably by humans.

This also solves the “problem” with the character encoding as JSON is by default encoded in UTF-8.

noxDB

On IBM i there are several ways to generate JSON. Personally I like the noxDB project. It is a fast and flexible JSON software library available as a ILE service program. It supports the data-gen opcode to easily generate JSON based on a data structure but also lets you create a fined tuned JSON document if you want to. So you have total control over the JSON generation. There are also some gems in the library like generating JSON based on a SQL result set.

The project comes in two flavors: version 1 = EBCDIC based , version 2 = UTF-8 based

The version 2 (UTF-8 based) is packaged as an RPM package and available in the RPM repository at RPG Next Gen. It can be installed with the iPKG client. The installation is as easy as always:

ipkg install noxdb2
ipkg install 'noxdb2-devel' loc('/home/mihael/include')

I won’t go into the details of how the noxDB service program can be used. There are dozens of examples.

stomp_command_send

We will rather take a look at the procedure that sends the data: stomp_command_send

dcl-pr stomp_command_send extproc(*dclcase);
    client pointer const;
    destination varchar(100) const;
    messageData pointer const options(*string);
    messageLength uns(10) const options(*omit : *nopass);
    contentType varchar(100) const options(*omit : *nopass);
    headers pointer const options(*nopass);
end-pr;

Client

As with most of the procedures we need to pass the client “object” (the pointer) to the procedure.

Destination

Second we need to tell the procedure to which queue the message should be sent. This name maps directly to a address name in Apache Artemis. In our case this is auction-events.

Message Data

We notice that the parameter for the message data is a pointer with options(*string). The procedure can work with a null-terminated string (meaning a string with the termination character x'00' at the end of it). It expects a null-terminated string if no message length is provided.

In RPG this means that we can declare a variable char or varchar and end the data with x'00'.

dcl-s data varchar(100) ccsid(*utf8);

data = '{ "id" : 358 }' + x'00';

stomp_command_send(client : 'auction-events' : %addr(data : *data));

But as options(*string) has been declared on the paramter we can also just pass the variable. No need to bother with any C-style string stuff.

dcl-s data varchar(100) ccsid(*utf8);

data = '{ "id" : 358 }';

stomp_command_send(client : 'auction-events' : data);

This looks much better. :)

It is also possible to pass a string literal.

stomp_command_send(client : 'auction-events' : '{ "id" : 358 }');

Message Length

Without passing the message length the procedure will try to determine the length by searching for the first x’00’ value. This is not always very convenient. If you want to send binary data and the data might have x'00' as a value then you need to pass the length of the data as a parameter.

Content Type

Similar to HTTP message we also should specify the Content-Type of the data we want to send so that the receiver knows how to interpret the data.

If we always want to the send the same type of data (meaning also the same Content-Type) then we can set the content type on the client and don’t have to pass it on every call.

stomp_setContentType(client : 'application/prs.rpgnextgen.auction')

The content type passed to the send procedure will overwrite any global settings on the client.

Headers

If we want some additional headers on the frame we send to the server we can pass them to the send procedure.

The headers parameter is declared as a pointer. This doesn’t tell us much. But it expects a list created with the Linked List service program. Each entry needs to be in the format of the data structure stomp_frame_header_t.

dcl-s headers pointer;
dcl-ds header likeds(stomp_frame_header_t) inz;

headers = list_create();
header.key = 'server';
header.value = 'ibmi.localhost';
list_add(headers : %addr(header) : %size(header));

...

stomp_command_send(client : 'auction-events' : %addr(auction) : %size(auction) :
    STOMP_MIME_BINARY : headers);

...

on-exit;
    list_dispose(headers);

Don’t forget to free the allocated memory of the list by calling list_dispose(headers).

Receipts

To make sure that the server has successfully received the message sent to the server we can request to acknowledge the receiving of a message with a receipt. So for every message we send to the server the server sends back a RECEIPT frame to tell the client that the message has been successfully received.

stomp_useReceipts(client : *on);

Note: Receipts only make sure that the server successfully received the message. It does not mean that the message has been successfully distributed to any other client.

Mark message as persistent

By default messages sent via the STOMP protocol are not persistent, meaning they will vanish if the server is restarted. STOMP also supports persistent / durable messages. This can be achieved by adding the header persistent to a message, see STOMP_OPTION_PERSISTENT which can be set on the STOMP client instance.

stomp_setPersistMessages(client : *on);

API Documentation

You can find the API documentation of the STOMP for ILE project at iledocs.rpgnextgen.com. For those interested in generating API documentation for ILE programs and service programs: The documentation at iledocs.rpgnextgen.com has been generated with the project iledocs-page-builder.

Happy integrating!

Mihael