RPG Next Gen

Batch - Consuming Web Services

Fri, 16 Jan 2026 00:00:00 +0100

Batch - Consuming Web Services

Most of the time the input for a batch workflow comes from databases or files. These are already very easily accessable through various libraries.

In the last years web services have gained much popularity and is one if not the most common way to exchange data. Most of the time the data is presented in JSON.

Object by Object

For very large sets of data it is very inconvenient to load the whole data set into memory before processing. I already made a post about how to process a JSON object by object by using the library Jackson, see Streaming data - the client side.

Authentication

Many web services are secured by something like OAuth 2.0 and OpenID Connect. OpenID Connect explicitly has a grant type for batch programs or M2M (machine to machine) communications:

Client Credentials

So an access token can be obtained from your OIDC provider just by passing a client id and a client secret. To make this as easy as it sounds I created a new project which contains an OidcTokenProvider. It just needs the configuration passed in the MicroProfile Config way like using a microprofile-config.properties file or using environment variables.

batch.ws.oidc.client-id
batch.ws.oidc.client-secret
batch.ws.oidc.scope (default: openid)
batch.ws.oidc.url (token URL)

Then the OidcTokenProvider can be injected and an access token obtained.

@Inject
private OidcTokenProvider tokenProvider;

...

String accessToken = tokenProvider.get();

Multiple Token Provider

The library provides a prepackaged token provider which can be used out-of-the-box (with the corresponding configuration). But sometimes we need to use multiple web services and each may have their own OIDC provider or client id. That means we need two instances of OidcTokenProvider with two different configurations.

This can be achieved with some lines of code. A new class needs to be created which extends AbstractOidcTokenProvider.

@ApplicationScoped
public class PackagingOidcTokenProvider extends AbstractOidcTokenProvider {

	@Inject
	public PackagingOidcTokenProvider(@ConfigProperties(prefix = "packaging") OidcTokenProviderConfig config) {
		super(config);
	}

}

By prefixing the config properties (prefix = "packaging") two seperate configuration instances can exist side by side. And instead of batch.ws.oidc.url the configuration packaging.oidc.url is used for this instance. And both token provider instances can be injected into the same bean.

@Inject
private OidcTokenProvider tokenProvider;

@Inject
private PackagingOidcTokenProvider packagingTokenProvider;

Source Code

All those classes have been grouped together into the library batch-ws.

<dependency>
	<groupId>com.rpgnextgen</groupId>
	<artifactId>batch-ws</artifactId>
	<version>[1.0.0,2.0.0)</version>
</dependency>

The source code is available at Bitbucket.

Wrap Up

Accessing web services in a batch prorgram can be very easy and even memory friendly. No need to dump the content to a local file and bother with CCSIDs and clean up strategies.

Happy coding!

Mihael

Batch Core - Dynamic Business Rules

Sun, 04 Jan 2026 00:00:00 +0100

Batch Core - Dynamic Business Rules

In this Java batch series I hardcoded the business rules in the program. This may be ok for some cases but in our fast moving world having the ability to change business rules without changing the program is a big plus.

And it is not only about having the end user change the business rules. I wouldn’t go that far in my first step. Having business rules extracted to a text file or a database is already a big plus because you can instantly change a business rule without too much fuzz.

Does the new or changed business rule needs some kind of testing?

Yes, definitely! But we don’t need to compile and test the whole program, just the business rule.

So how to implement that?

Implementation

Having such a mature programming language as Java has its advantages. There probably is already a solution to your problem as most developers don’t face brand new problems. Or there is a solution which can be modified so it fits your problem.

And that is also the case here. Dynamic business rules can be easily implemented using the Jakarta Expression Language. Most have heard of Jakarta EL (Expression Language) in conjunction with Java Server Faces and Java Server Pages. But it can also be used stand alone, outside of Jakarta EE servers.

Eclipse Expressly is the reference implementation of the Jakarta EL specs. And if we are taking a look at the dependencies of Expressly we astonishingly see only a single dependency – the EL API. That means we can use Expressly in many scenarios and environments.

User Defined Context

Mostly there are two ways to use Jakarta EL.

evaluate an expression and use the result of the evaluation for further processing
execute an expression and do every processing inside the expression

If we evaluate an expression and work with the result that would mean that we need to hard code what to do with the result. That would remove some of the dynamics we get from using an expression language.

The solution is … providing your own context (or rather your own ELResolver class). That means we can add our own objects to the expression context so that our own objects are available in the expression and thus can be queried and changed (despite ChatGPT saying that this is not possible, so much for our glorious AIs 😄).

ChatGPTs answer:

❌ Not possible in EL

You cannot do this:

loop + modify entity field

assign values in EL

create mutable counters in EL

because EL has no assignment operators.

But I can do this:

1	entity.packagingWorkplace = entity.line == 5 ? 5 : entity.packagingWorkplace

… which assigns a value to a field by using an operator. AI is so ridiculous.

Business Rules Expressions

Our initial business rules from the first Java Batch post were the following:

fragile product → workplace 4
oversize product → workplace 3
line 5 (hall 2) → workplace 5
else → distribute evenly between workplace 1 and 2

Rule 1 to 3 are very easily translated to the following Jakarta EL expressions:

1
2
3

entity.packagingWorkplace = entity.fragile ? 4 : entity.packagingWorkplace
entity.packagingWorkplace = entity.volume >= 120000000 ? 3 : entity.packagingWorkplace
entity.packagingWorkplace = entity.line == 5 ? 5 : entity.packagingWorkplace

But the fourth rule is a bit more complicated as we need to save some state (the last workplace assignment for that case). This can be solved in various ways. I chose to add a counter object to the context.

private Map<Object, Object> userContext = new HashMap<>();
private AtomicInteger counter = new AtomicInteger();

...

elResolver = new DelegatingMapELResolver(userContext);
elContext.addELResolver(elResolver);

...

userContext.put("counter", counter);

The following expression queries and increments the counter:

1	entity.packagingWorkplace = ((counter.getAndIncrement() mod 2) + 1) : entity.packagingWorkplace")

But we only want to use this expression as the last resort if no other expression matched.

1	entity.packagingWorkplace = entity.packagingWorkplace == NULL ? ((counter.getAndIncrement() mod 2) + 1) : entity.packagingWorkplace")

Note: Using the ternary operator “?” is a workaround for the lack of an if/else construct in the Jakarta Expression Language.

Performance

I was positively surprised by the performance of Expressly. I had a database table with around 850k records. I read the whole table and executed a single expression on every record. The whole processing was done in under 15 seconds. Kudos to the developers of Jakarta EL and Expressly.

Source Code

I have uploaded an example project at Bitbucket. Feel free to take a look at it. I added various branches to the Git repository which reflect my various steps and stops of my journey.

Wrap Up

Turning the program from hardcoded business rules into using configurable user defined business rules was much easier than anticipated and the real surprise for me was the very good performance. Again it shows that Java is a really great programming language with its excellent libraries. One can achieve so much with so little effort.

Happy new year! 🎉

Mihael

Batch Core - Maintainability

Wed, 31 Dec 2025 00:00:00 +0100

Batch Core - Maintainability

So how easy is it to maintain and extend a project which uses batch-core. Of course that mainly depends on the change. But some frameworks and libraries make it easier and some harder to integrate changes.

We take our production planning example and want to only process the data for a specific production day.

So what do we need to do:

pass the desired production day to the program (command line parameter)
get the passed date to the step implementation
change the SQL to optionally use the passed date

Changes

The production day can be passed to the program in various ways as we are using MicroProfile Config and also have our own ConfigSource for the program call parameters.

1	java -cp production-planning.jar:libs/* rpgnextgen.batch.Main start packaging --production-day 2025-12-30

In the step implementation we need to add the configuration property.

1
2
3

@Inject
@ConfigProperty(name = "production-day")
private Optional<LocalDate> productionDay;

We make this an Optional because we also want to be able to not pass a production day. And as we are using the SmallRye implementation of MicroProfile Config we can directly use LocalDate even though the MicroProfile Config spec doesn’t support it.

The SQL only needs a little tweaking in the WHERE clause to support optionally passing a production day. If none is passed the parameter is expected to be null.

@SqlQuery("""
    SELECT id, itemId, productionDay, category, line, width, height, depth, fragile
    FROM productionPlanning
    WHERE (:productionDay IS NULL OR productionDay = :productionDay)
    """)
@RegisterBeanMapper(ProductionPlanningEntry.class)
Stream<ProductionPlanningEntry> stream(@Bind("productionDay") LocalDate productionDay);

Now we can either pass the production day as a LocalDate object to the stream method or pass null.

1	dao.stream(productionDay.orElse(null))

And that’s it! 🎉

Wrap up

batch-core did not stand in our way in this change but really helped us. The production day can be easily added as a parameter to program call (but also as an environment variable as MicroProfile Config is used). Injecting the production day to the step implementation was also easy and the adjustment in the SQL had nothing to do with batch-core.

What we didn’t cover was parameter validation. Passing a not valid date results in an exception and abnormal termination. That may be a desired result but it depends on your environment.

So this was a really easy one 😀 .

Happy coding!

Mihael

Batch Core - Minimalistic Example

Tue, 30 Dec 2025 00:00:00 +0100

Batch Core

I decided to extract the core classes of my last Java batch example and put them in their own project. The project is uploaded to Maven Central. There are two versions of batch-core:

version 1.x.x = JBoss Weld 4.x
version 2.x.x = JBoss Weld 6.x

So far I mostly used Weld 4 but as I have no conflicting dependencies I wanted to give Weld 6 a try.

<dependency>
	<groupId>com.rpgnextgen</groupId>
	<artifactId>batch-core</artifactId>
	<version>2.0.0</version>
</dependen

Most simple example

The simplest example is to just implement WorkflowStep and be done.

@ApplicationScoped
@Named("hello")
public class HelloStep implements TypedWorkflowStep<Void> {

	@Override
	public Void execute(Void parameter) {
		System.out.println("Hello World");

		return null;
	}

}

That’s it! No other classes need. No other files needed (besides those for CDI support).

Note: Don’t forget to setup your project for CDI because your step implementation needs to be a bean.

More icing on the cake

And as the step implementation is a bean I can inject a configuration value directly into the bean without any fuzz.

@ApplicationScoped
@Named("hello")
public class HelloStep implements TypedWorkflowStep<Void> {

	@Inject
	@ConfigProperty(name = "name", defaultValue = "World")
	private String name;

	@Override
	public Void execute(Void parameter) {
		System.out.println("Hello " + name);

		return null;
	}

}

… or I could inject other beans here. Feel free! :)

Happy batching!

Mihael

JBatch Streams - Refined!

Sun, 28 Dec 2025 00:00:00 +0100

JBatch Streams - Refined!

The last post was about the low cost alternative Java Streams for batch processing. But the stream processing was pressed into a Jakarta EE Batchlet which has some nice side benefits but also some overhead which I would like to get rid off in this post.

Kick Jakarta EE Batch

First I kicked out Jakarta EE Batch. This was not too difficult and in hindsight the number of removed libraries and dependencies was surprisingly small.

Replacements

But removing the dependencies from the Maven POM was the least effort in this milestone. I now had to replace some of the functionality Jakarta EE Batch brings to the table.

Batch Properties Injection
Workflow Configuration
Workflow Starter

Batch Properties Injection

Instead of having a separate mechanism for injecting properties I used MicroProfile Config for injecting values. I added a new ConfigSource implementation which delivers the values passed as batch properties via the command line. The only thing one needs to take care of is using a Provider instead of the concrete type because at the time the ConfigSource instance is created the command line parameters haven’t been evaluated yet.

1
2
3

@Inject
@ConfigProperty(name = "database")
private Provider<String> database;

As MicroProfile Config is used for providing injectable values I can override the parameters passed on the command line with values from a ConfigSource rated higher than the one for the batch properties.

The ordering is as follows:

100 : microprofile-config.properties file
200 : Batch properties
300 : Environment variables
400 : Java system properties

So if every ConfigSource provides a value for the same property the batch properties passed on the command line would trump the value in the microprofile-config.properties file. The value from the environment variables ConfigSource would trump the batch properties and the Java system property would trump them all.

Workflow Configuration

Jakarta EE Batch supports many batch scenarios and features, but I only intend to support a simple batch job scenario and don’t need all those additional capabilities.

The result is a workflow which declares an array of steps that should be executed. So the developer only needs to implements two interfaces: Workflow and WorkflowStep. All declared workflow steps will be executed sequentially in the specified order.

Optionally you can also implement only the WorkflowStep interface if you just only have one step anyway. No need to also implement a Workflow.

Workflow Starter

Jakarta EE Batch uses XML to define the workflow and also the starting point of the workflow. In this little batch implementation we are using CDI to mark possible starting points with the Named annotation.

@ApplicationScoped
@Named("packaging")
public class PackagingWorkflow implements TypedWorkflow<Void> {

  	@SuppressWarnings("unchecked")
  	@Override
  	public Class<? extends TypedWorkflowStep<Void>>[] steps() {
    		return new Class[] { DatabaseInitializer.class, PackagingWorkplaceSelectionStep.class };
  	}

}

The workflow name packaging is passed on the command line as a second parameter to the application.

1	java -cp production-planning.jar:libs/* rpgnextgen.batch.Main start packaging --database h2

The batch parameters/properties are passed as additional parameters. The parameter name is prefixed with two hyphens followed by the parameter value.

The workflow name can also be prefixed with workflow: which identifies it as a workflow name. We can also directly start a specific step by providing a step name prefixed with step:

1	java -cp production-planning.jar:libs/* rpgnextgen.batch.Main start step:packaging-workflow-selection

The java class for the step must be annotated with Named and it must also be available as a CDI bean.

@ApplicationScoped
@Named("packaging-workplace-selection")
public class PackagingWorkplaceSelectionStep implements TypedWorkflowStep<Void> {
    ...
}

By using CDI and Named we can package multiple workflows into one application and choose the workflow at application start by passing the corresponding workflow or step name.

Dependencies

We only declared 10 dependencies and many of those dependencies are about the database and with all the transitive dependencies we are at 32 libraries in total. That is very maintainable compared to other programming languages where even for the simplest things you got 100(s) of dependencies.

And the maintainers of the dependencies …?

Apache Software Foundation
Eclipse Foundation
IBM
JBoss (Red Hat)
OW2 (Non Profit Community)
SmallRye (Red Hat)

There are only 2 libraries which doesn’t belong to any big organization:

Jdbi
GeanTyRef

Compare that to the dependency graph of other programming languages.

Source Code

The source code for this example is available at Bitbucket.

Wrap Up

What has been achieved by removing Jakarta EE Batch:

more generic approach to batch processing
more flexibility
better runtime performance
chaining
less dependencies

Is it a replacement for Jakarta EE Batch?

Definitely not! Jakarta EE Batch brings so much to the table that it is by far no general replacement for it.

But if you want to develop a small batch programm in Java and you don’t need all the bells and whistles of Jakarta EE Batch then it may be a good alternative.

Happy streaming!

Mihael

JBatch Alternative on IBM i

Sun, 14 Dec 2025 00:00:00 +0100

JBatch Alternative on IBM i

JBatch (or rather Jakarta EE Batch) is a nice and structured way of implementing a batch job. It has many features like partitioning a task, dynamic job flow, persisting job execution, restarting a job, … .

These are great features but not every batch job needs those features. From time to time you just need to process a database table. No need for partitioning, dynamic job flow or persisting the job execution. If the job fails you just run it again (after solving the problem ;-) ).

In those cases Jakarta EE Batch would be overkill.

The Alternative

Often the solution is already there. Ready to be discovered and used.

Java Streams

Stream from the java.util.stream package is an excellent lightweight alternative. It can also be integrated into a Jakarta EE Batch program (by using a stream in a Batchlet).

But JBatch and Streams are fundamentally different!

Depending on the perspective … yes, they are different. But they also try to achieve the same thing: Run a series of steps to process data.

A fundamental thing in Java Streams is the concept of data flowing - handed from one execution step to the next. That is a really nice concept. Jakarta EE Batch follows a different concept: Input -> Processing -> Output

The sad thing is that you can only have one processor in this chain of steps.

Only a single processor element may be specified. – Jakarta EE Batch Spec 2.0

It would be nice if you could chain multiple processors and pass the data from one processor to the next.

And that is where streams are much more versatile whereas Jakarta EE Batch has a rather rigid approach.

When using streams we need to deal with functions. I won’t deep dive into functions here as it is a really big topic. We will just scratch those parts which matter for this example.

Our Use Case

Our example will be straight from the manufacturing department. We are producing some goods and these goods need to be packaged. We have several lines and workplaces for packaging and depending on the type and size of the product we need to assign it to the corresponding workplace.

We have a single table as input which contains all manufactured goods for the next production days and we have another table which receives the output data.

Rules

We have five workplaces for packaging.

Workplace 1 : normal packaging (hall 1)
Workplace 2 : normal packaging (hall 1)
Workplace 3 : oversize packaging (hall 1)
Workpalce 4 : fragile packaging (hall 1)
Workplace 5 : packaging for production hall 2

So not all goods can be packaged at every workplace. We need to setup rules by which the goods are assigned to the corresponding workplaces.

fragile product → workplace 4
oversize product → workplace 3
line 5 (hall 2) → workplace 5
else → distribute evenly between workplace 1 and 2

Input

In Jakarta EE Batch we would implement an ItemReader. When using streams the input comes from a java.util.stream.Stream implementation. Luckily we won’t have to implement this ourselves.

We will be using the Jdbi project for reading the data from the database. Jdbi provides a nice feature where you can more or less just write the SQL and it will make a nice POJO from the data. If we want to fetch a set of data we would just define a java.util.List as the return type and we would get all the data in one step. That may be convenient with small sets of data but when you are processing millions of rows you are running out of memory faster than you can count to three.

The solution for this is to use java.util.stream.Stream as the return type. Using a Stream will only load the next row of data into memory and is much better when it comes to conserving resources.

@SQLQuery("""
    SELECT id, itemId, productionDay, category, line, width, height, depth, fragile
    FROM productionPlanning
""")
@RegisterBeanMapper(ProductionPlanEntry.class)
Stream<ProductionPlanEntry> listPlannedProduction();

Processor

We got the stream we want to operate on. The main methods we need from a Stream object are map, filter and collect.

filter : We get an object of class x. Our function determines if the object will be dropped from the stream. Just return false for dropping the object from the stream.
map : We get an object of class x. We can modify the received object to our liking. Our function can freely decide what to return. It can be the same object, another object (even of another class) or null.
collect : This is a terminal operation which means that this function will receive all objects from the stream and process them in this final step. There is no further processing.

We can create a function for each rule we want to implement. But as the rules are very miniscule we just put them all into one function. We can use anonymous functions or implement them as a class.

There are multiple Java interface for functions. Here are the most common ones:

java.util.function.Function : Accepts an object and returns an object. The passed and returned object doesn’t need to be the same object or even of the same class. This can be used by the map method of a Stream.
java.util.function.Predicate : Accepts an object and returns a boolean result. This can be used by the filter method of a Stream.

There are many more interface (like Consumer, Supplier, …) but those are of no interest to us for this example.

public class PackagingWorkplaceSelector implements Function<ProductionPlanningEntry, ProductionPlanningEntry> {

	private final long OVERSIZED = 120000000l; // volume in millimeter³
	private int lastDefaultPackagingWorkplace = 0;

	@Override
	public ProductionPlanningEntry apply(ProductionPlanningEntry entry) {

		if (entry.isFragile())
			entry.setPackagingWorkplace(4);
		else if (isOversized(entry))
			entry.setPackagingWorkplace(3);
		else if (entry.getLine() == 5)
			entry.setPackagingWorkplace(5);
		else
			entry.setPackagingWorkplace(getNextDefaultWorkplace());
		return entry;
	}

	private Integer getNextDefaultWorkplace() {
		lastDefaultPackagingWorkplace = lastDefaultPackagingWorkplace == 1 ? 2 : 1;
		return lastDefaultPackagingWorkplace;
	}

	private boolean isOversized(ProductionPlanningEntry entry) {
		return entry.getVolume() >= OVERSIZED;
	}

}

Output

Writing the data to a file or table can be done in a mapping function. We use Jdbi for writing the data to an output table.

@SqlUpdate("""
    INSERT INTO productionPlanningPackaging (itemId, productionDay, packagingWorkplace)
    VALUES(:itemId, :productionDay, :packagingWorkplace)
    """)
void insertPackagingWorkplace(@BindBean ProductionPlanningEntry entry);

Putting it all together

Putting all those parts together may look like this:

public class ProductionWorkflow implements Batchlet {

	private final Logger logger = Logger.getLogger(this.getClass().getName());

	@Inject
	@BatchProperty(name = "database")
	public String database;

	@Override
	public String process() throws Exception {
		Jdbi jdbi = CDI.current().select(Jdbi.class, NamedLiteral.of(database)).get();

		try (Handle handle = jdbi.open()) {
			ProductionPlanningDao dao = handle.attach(ProductionPlanningDao.class);
			Map<Integer, List<ProductionPlanningEntry>> groupedWorkplaces = dao.stream()
                    .map(new VolumeCalculator())
                    .map(new PackagingWorkplaceSelector())
                    .map(new PackagingWorkplaceWriter(dao))
                    .collect(Collectors.groupingBy(ProductionPlanningEntry::getPackagingWorkplace));

            for (Integer key : groupedWorkplaces.keySet()) {
                logger.info(key + ": " + groupedWorkplaces.get(key).size());
            }
		}

		return "ok";
	}

	@Override
	public void stop() throws Exception {
      // nothing to do
	}

}

Output:

2025-12-22 10:48:02 INFORMATION rpgnextgen.packaging.ProductionWorkflow: 1: 427
2025-12-22 10:48:02 INFORMATION rpgnextgen.packaging.ProductionWorkflow: 2: 426
2025-12-22 10:48:02 INFORMATION rpgnextgen.packaging.ProductionWorkflow: 3: 36
2025-12-22 10:48:02 INFORMATION rpgnextgen.packaging.ProductionWorkflow: 4: 100
2025-12-22 10:48:02 INFORMATION rpgnextgen.packaging.ProductionWorkflow: 5: 11

I planned to support the use of different databases in this demo and thus the database gets injected, see batch property database. Depending on the database we get the corresponding Jdbi provider.

The different functions are chained in the stream and as a final step the number of products for each workplace are summed up.

Miscellaneous

Sometimes we have processed all objects in the stream in a filter or mapping function and just want to just drop all objects after our last function. filter and map are intermediate operations. We need to make a final call on the stream with a terminal operation. There is no terminal operation like sink, drop or dump but you can use count which is probably the fastest terminal operation.

If we want to make sure that no null values are passed down the stream we can use stream.filter(Objects::notNull) to filter out all null values.

Source Code

The source code for this example is available at Bitbucket.

Wrap Up

The JBatch example project was a good starting point for using streams in a batch process. We get a lot of benefits from it like CDI and argument processing and it works but we could make it even easier and more straightforward if we drop the Jakarta EE stuff and just use the streams (and perhaps Weld for CDI ;-) ). This sounds like a topic for the next blog post ;-).

… and if you are interested in using streams in RPG there is actually an implementation at Bitbucket. I have written a post about it a while ago.

Happy streaming!

Mihael

QR Codes on IBM i

Sun, 29 Jun 2025 00:00:00 +0200

QR Codes on IBM i

QR Codes are a very convenient solution for many use cases. In contrast to barcodes a very broad range of data can be encoded into a QR Code. Today you will encounter many QR codes which have a URL encoded into it which can be scanned with your smartphone to open a web site.

QR codes are 2 dimensional in constrast to barcodes which just use one dimension.

Another difference to barcodes is the size of a QR codes. The size of a QR code depends on some factors:

Size of encoded data
Type of encoded data
Level or error correction

For a better explaination and more details on this see Information capacity and versions of the QR Code.

And if this is not enough motivation to learn something about QR codes and how to create them on IBM i natively … have you ever heard of the “GS1 Sunrise 2027 Initiative”?

Transitioning to 2D barcodes for POS is a multi-step process. The GS1 US Sunrise 2027 initiative lays out a plan to help ensure 2D barcodes will be scanned and processed at any retail POS by the end of 2027.

Quote from the GS1 website.

And by 2D barcodes they mean QR codes and data matrix.

Native Support

There is native support to print QR codes on IBM i. But this doesn’t help if you need to output the QR code to another medium like the web or a stream file.

There are solutions but none of them are freely available and directly usable from the ILE environment. Most incorporate other programming languages like Python or Node.js.

And don’t even start with any kind of printer files … yak.

But a free ILE solution is not that hard to implement. There are a lot open source libraries available on the internet which can help in generating a QR code and can be used in the ILE environment.

The Steps

Generating a QR code is more or less done in two steps.

calculate which blocks needed to be black / white
draw the blocks (on a canvas like the browser window or into a stream file)

For step 1 we use the C library libqrencode from Kentaro Fukuchi available as an open source library on GitHub.

The library is mostly C89 compatible and therefore can be compiled as ILE modules and bound to an ILE service program. There is a precompiled version available in the RPM repository at repo.rpgnextgen.com and installable via iPKG.

1	ipkg install libqrencode

The RPG prototypes can also be installed with iPKG.

1	ipkg install 'libqrencode-devel' loc('/home/mihael/include')

Getting the first step done in an RPG program is not that difficult.

dcl-proc main;
    dcl-ds qrCode likeds(qrcode_t) based(qrCodePtr);
    dcl-s input varchar(100) ccsid(*UTF8);

    input = 'Mihael Schmidt';

    qrCodePtr = QRcode_encodeData(%len(input) : %addr(input : *data) : 0 : 0);

    on-exit;
        QRcode_free(qrCodePtr);
end-proc;

The C function QRcode_encodeData takes some raw data and encodes it into the QR code. For that it needs to know how large the data is (size) and where data starts (address of the variable, data part).

The parameter 3 and 4 are QR code specific. The version (parameter 3) defines the number of “modules” used in the QR code. Modules are the blocks in the QR code. More modules = more data can be encoded. The value 0 means the minimum number of modules for the passed data will be used.

Parameter 4 - ecLevel - defines the level or error correction. The higher the level the less data can be encoded into QR code with the same version. The available values range from 0 to 3 where 0 is the lowest level of error correction and 3 the highest.

The library libqrencode offers many more functions to encode data but this is the easiest one to use.

Output on Screen

We can output the data on the screen with just some lines of code. We take the variable qrCode from the previous block of code and pass it to the procedure drawOnConsole. The filled modules will be drawn as an X.

dcl-proc drawOnConsole;
    dcl-pi *n;
        qrCode likeds(qrcode_t);
    end-pi;

    dcl-s line varchar(50);
    dcl-s x int(10);
    dcl-s y int(10);
    dcl-s qrBlockData char(1) based(qrBlockPtr);

    qrBlockPtr = qrCode.data;

    for x = 1 to qrCode.width;
        clear line;

        for y = 1 to qrcode.width;
            if (%bitand(qrBlockData : QRCODE_BLACK) = QRCODE_BLACK);
                line += 'X';
            else;
                line += ' ';
            endif;
            qrBlockPtr += 1;
        endfor;

        dsply line;
    endfor;
end-proc;

The returned QR code data from QRcode_encodeData is really just a matrix with some bit flags switched on or off, f. e. for the fill state of a block.

From the libqrencode documentation:

// MSB 76543210 LSB
//     |||||||`- 1=black/0=white
//     ||||||`-- 1=ecc/0=data code area
//     |||||`--- format information
//     ||||`---- version information
//     |||`----- timing pattern
//     ||`------ alignment pattern
//     |`------- finder pattern and separator
//     `-------- non-data modules (format, timing, etc.)

To test if a block is filled (black) or not we just need to define some constants and test it with the bitand built-in function. The QR code data structure contains the used width which is the number of modules in one row. The rows of the QR code are contained in the data subfield of the data structure. The data of each row is immediately followed by the data of the next row. There is no separator. But we got the width of each row so we can compute the start and end of each row.

To get the data for each module (block) we kind of dynamically overlay the QR code data with a character field with a length of just 1.

qrCode.data is just a pointer to the data and not the data itself. By assigning the data pointer to the pointer the char(1) field is based on the field holds the first module data. And just by incrementing this pointer (qrBlockPtr) by one we can move the field to the next module information.

The output looks something like this:

DSPLY  XXXXXXX   X X XXXXXXX
DSPLY  X     X X X X X     X
DSPLY  X XXX X X XX  X XXX X
DSPLY  X XXX X     X X XXX X
DSPLY  X XXX X XXX X X XXX X
DSPLY  X     X XXX   X     X
DSPLY  XXXXXXX X X X XXXXXXX
DSPLY          X  X
DSPLY  XX X  XX  X X XXX XX
DSPLY   X X   XX X XX   XXXX
DSPLY      XXXXXXXXX  X XX X
DSPLY  X   XX X  X   XX X  X
DSPLY  XX X XXX   XX   X  X
DSPLY          X  XX  X XX
DSPLY  XXXXXXX X X XXX XXXX
DSPLY  X     X        X   XX
DSPLY  X XXX X  XX  X XX
DSPLY  X XXX X XXX XX XX  XX
DSPLY  X XXX X  XXXX XXXXX X
DSPLY  X     X XX   X  X
DSPLY  XXXXXXX XX    XXXX X

This is a QR code with 21 x 21 modules (blocks) and equals the QR code version 0.

Not too bad :)

Output to Image

But in the modern world we seldom need to output a QR codes onto a console screen. But what is much more common is the output to an image file f. e. in PNG format.

There is no support out-of-the-box available in the ILE environment for dynamically creating PNG images so we need to extend our toolbox by another software library.

libpng is a good candidate as it boasts to be written in ANSI C (C89) but the project is quite big and I only need to output some black and white blocks. So I keep looking for another candidate.

lodepng is a very small library which can be easily embedded in a project as it only consists of a single header file and a single source file.

In addition to C++, LodePNG also supports ANSI C (C89), with all the same functionality: C++ only adds extra convenience API.

That sounds promising!

And the project really delivered! It hasn’t promised too much. Though I had to work around some CCSID problems (as always ;-), it compiled out of the box (not counting the CCSID problems) and we can install it and the corresponding copybooks with iPKG.

1 2	ipkg install lodepng ipkg install 'lodepng-devel' loc('/home/mihael/include')

Note: You can also compile all these software libraries by yourself and don’t have to use iPKG. You can find the libraries with their Makefile at my Bitbucket page.

So far we can create the QR code data and know which blocks are filled and which are not filled. Now we need to output that data to a PNG image.

First we need to convert the QR code data to pixel data. The raw pixel data is a block of memory where each pixel consists of 3 or 4 values (RBG - Red Green Blue or RBGA - Red Green Blue Alpha). Each value needs to hold a number from 0 to 255. We declare this as uns(3) – a one byte unsigned integer. So our block of memory has the size of image width x image height x 4 (we use RGBA).

Now we just need to loop through our QR code data and create pixel data from each module data (which is too lengthy to show here). The raw pixel data can now be passed to a C function from the lodepng project.

1	lodepng_encode_file(filename : pixels : imageWidth : imageHeight : 6 : 8);

Most parameters are self describing. Only the 5th and 6th parameter needs some explaination.

The 5th parameter defines the color type used in the image, 6 = RGBA.

The 6th parameter defines the color depth in bit. The range of values depend on the used color type. Valid values for the color type RBGA are either 8 or 16.

This whole process has been wrapped into a nice procedure in my project qrtools which can be installed via iPKG.

1 2	ipkg install qrtools ipkg install 'qrtools-devel' loc('/home/mihael/include')

And the usage of the procedure looks like this.

dcl-proc main;
    dcl-s input varchar(100) ccsid(*utf8);

    input = 'Mihael Schmidt';

    qrtools_toPng('/home/mihael/qrcode.png' : %addr(input : *data) : %len(input) : 200);
end-proc;

In this example the created image has a size of 200 x 200 px. The QR code is automatically scaled to the maximum size inside the given image dimension. Any not used space in the image will be drawn as 100% transparent pixels.

Easy peasy! ;-)

Wrap Up

Now we can natively create QR codes in our beloved ILE environment without having to switch to any environment which is not integrated as well as our polyglot environment.

Currently the project qrtools covers only the creation of a PNG image. Possibly other output formats could be SVG, EPS, JPEG. This might get implemented depending on the feedback.

Demand determines supply. ;-)

Happy QR (en)coding!

Mihael

ILEDocs ... the 3rd take ... action!

Tue, 15 Apr 2025 00:00:00 +0200

ILEDocs … the 3rd take … action!

ILEDocs is now in its third version and implementation. It started in RPG then transitioned to Java and now arrived in the Node.js world. It differentiates from its predecessor in that it now produces static HTML documents (again) instead of being a dynamic web application.

So what does it do and what has changed?

It still does what it should do: producing an API documentation of the programs and service programs, be it RPG or CL.

You still comment a program, module or procedure with a comment block like this:

///
// List order entries
//
// Returns all order entries for the specified order.
//
// @param Order number
// @return List of order entries, see order_entry_t
///

And the output is still HTML pages.

Though there are some new features added in this implementation.

New Features

No extra software required: Unlike the Java version, it doesn’t need any additional software — no database or application server. Just the program and the source code.
Cross-platform compatibility: It doesn’t have to run on IBM i, though it can, and doing so may have some advantages.
Multiple page generation: You can generate multiple HTML pages from a single source code base.
Optional overview page: If the content is split across multiple HTML pages, an overview page can optionally be generated.
JSON output: A JSON document containing the generated pages can be created for further programmatic processing.
Custom templates: The HTML templates can easily be replaced with your own custom set.

Requirements

As this is a Node.js application you need to install the Node.js runtime (version >= 14).

1	yum install nodejs20

Installation

The next step is to download the repository of the project into the IFS or clone the repository in the IFS.

To install the dependencies of this Node.js applications you need to execute

1	npm install

After having the dependencies installed the application can be built.

1	npm run build

Configuration

The configuration file defines what will be parsed and how/where the output will be generated. The format of the file is JSON.

{
  "project": "project name",
  "title": "title used in templates",
  "output": {
    "path": "path to the output directory",
    "suffix": "suffix of the rendered pages, f. e. html"
  },
  "templates": { "path": "path to the templates directory" },
  "overview": ["index", "index_json"],
  "source": "path to the RPG or CL source code stream file",
  "section": []
}

Section

Each section can be configured as follows:

{
    "id" : "internal section id",
    "name" : "section name used in template",
    "startsWith" : "including symbols starting with this string (optional)",
    "symbols" : "symbols included by name (optional)",
    "default" : true | false,
    "order": pageOrder,
    "descriptionAsHeader" : true | false,
    "filename" : "page output filename"
}

If the project has defined a default section every symbol which cannot be mapped to a section will be mapped to the default section.

The order attribute defines the order/index in which the section is listed in the overview page.

Each page may have a block of text at the head of the page. If the text should be the ILEDocs description from the module then set descriptionAsHeader to true in the section config.

Start

The application can be started with npm start.

The project configuration file can be passed as a parameter to npm start:

1	npm start project-config-myapp.json

By default the output will be placed into the folder pages if not configured otherwise.

Debugging

The ILEDocs Page Builder project uses the debug package. To get the debugging output you just need to activate it by setting the environment variable DEBUG to iledocs:*.

1	export DEBUG=iledocs:*

or in Powershell

1	$env:DEBUG="iledocs:*"

To enable debugging output for single components see the list of defined logging namespaces for the application in the README of the project.

Example

The JSON and XML library noxDB is a great example. The prototypes, constants and data structures of this project are all placed in a single stream file. We split the content up into multiple pages.

Basics
Constants
Variables
Serializer
SQL
Generator and Parser

For each page the associated exported symbols which are going to go into that page are defined like this

{
  "id": "variables",
  "name": "Variables",
  "symbols": ["jx_DelimiterDS", "jx_iterator"]
}

or we can select all exported symbols which start with a given string:

{
  "id": "constants",
  "name": "Constants",
  "startsWith": "JX_",
  "order": 2
}

To add an overview page to the generation process we add the configuration entry overview as a top level attribute in the configuration JSON:

1	"overview" : ["index", "page"]

Note: You can see the full configuration file for noxDB at the ILEDocs Page Builder repository.

Limitations

Currently only a single stream file is parsed for generating the documentation. It would be nice to parse multiple stream files in the process.

Real World Example

All documentation available on iledocs.rpgnextgen.com has been generated by ILEDocs.

Wrap Up

It has never been easier to document programs and service programs and integrate the process into a build pipeline which also makes the publishing of the documentation very easy.

For any further information on ILEDocs Page Builder please take a look at README file in the project repository.

Happy documenting!

Mihael

Jdbi, Streaming and Row Reducer

Sun, 19 Jan 2025 00:00:00 +0100

Streaming data from a Java web service with JAX-RS is pretty straight forward as we have seen in one of my previous posts. We can utilize the java.util.stream.Stream interface and its implementations for this.

And streaming data from a database with Jdbi is also easy by returning the values as a ResultIterable.

But what if we want to group/reduce the data? Can we use the row reducer from Jdbi?

No. The row reducer will read all data rows and store the result in a map. This may consume a ton of memory and may result in killing our application by running out of memory. So no more streaming when using a row reducer.

Solution

We should

get a Stream instance from the ResultIterable object (returned from the SQL Object interface)
not call any terminal operation method on the stream which would result in reading all the data and thus again ending streaming the data
prepare the order of the data so that we can identify when a group level break/control break occurs

We cannot use the annotation UseRowReducer from Jdbi for this as this would result in reading all the data and terminates streaming. We need to do the row reducing by ourselves.

Row Reducer

But what exactly is row reducing?

It is exactly what is says: reducing rows

Most of the time we have some “master” data row which gets some extra columns from some “detail” data, f. e. an order (master) with many items (details).

M1a  |  M1b  |  M1c  |  D1a  | D1b
M1a  |  M1b  |  M1c  |  D2a  | D2b
M1a  |  M1b  |  M1c  |  D3a  | D3b
M1a  |  M1b  |  M1c  |  D4a  | D4b
M2a  |  M2b  |  M3c  |  D8a  | D8b
M2a  |  M2b  |  M3c  |  D9a  | D9b

Here we have two master data entries where one master data entry (M1) has four detail entries and the other (M2) has two detail entries.

For M1 we need to reduce it to one object which has a list with 4 child objects. So when the key of the master data changes we need to make a “break”, process the master data and its details data and then start with the next master data entry.

This can be done with implementing the Jbdi RowMapper interface.

public class CustomerControlBreakMapper implements RowMapper<Customer> {

    private Customer customer;
    private Customer returnedCustomer;

    private RowMapper<Customer> customerMapper = BeanMapper.of(Customer.class, "c");
    private RowMapper<Email> emailMapper = BeanMapper.of(Email.class, "e");
    private RowMapper<EmailType> emailTypeMapper = BeanMapper.of(EmailType.class, "et");

    @Override
    public Customer map(ResultSet rs, StatementContext ctx) throws SQLException {
        if (customer == null || !customer.getId().equals(rs.getInt("c_id"))) {
            returnedCustomer = customer;

            customer = customerMapper.map(rs, ctx);
        } else {
            returnedCustomer = null;
        }

        Email email = emailMapper.map(rs, ctx);
        email.setEmailType(emailTypeMapper.map(rs, ctx));
        customer.addEmail(email);

        return returnedCustomer;
    }

}

With this implementation we get a Customer object with its e-mail addresses returned on each control break. On any other row we just get null (which we can later filter out with the java.util.stream.Stream::filter method). So in the end we just get our customer objects (with the e-mail address objects) in the stream.

Putting it all together

public Pair<Stream<Customer>, Handle> list() {

    // get the database connection
    Handle handle = jdbi.open();

    CustomerControlBreakMapper mapper = new CustomerControlBreakMapper();
    handle.registerRowMapper(mapper);

    // use SQLObjects for executing the SQL
    CustomerDao dao = handle.attach(CustomerDao.class);

    // get the stream from the ResultIterable
    Stream<Customer> customerStream = dao.list().stream();

    return Pair.of(stream, handle);

}

That looks pretty manageable but …

Hmmm … the last group of data is missing in the result!

As with any control break logic we need to adjust the code a bit because there is no control break after the last group of data rows.

The Final Group

The code is almost complete as it is because we already have processed every data row and have the last customer object with all the e-mail addresses. It just isn’t returned as there is no control break after the last row.

The variable customer in the CustomerControlBreakMapper class already holds the final customer object ready to be returned. We just need to get it somehow.

For that we introduce a new interface:

import java.util.Objects;
import java.util.stream.Stream;

public interface ControlBreakSupport<T> {

    T getFinalGroup();

    default Stream<T> stream() {
        return Stream.generate(this::getFinalGroup).limit(1).filter(Objects::nonNull);
    }

}

This interface lets you create a new stream of the final group (customer) object which we can concat with the Stream instance we get from the ResultIterable.

Let’s do it!

public class CustomerControlBreakMapper implements RowMapper<Customer>, ControlBreakSupport<Customer> {

    private Customer customer;
    private Customer returnedCustomer;

    private RowMapper<Customer> customerMapper = BeanMapper.of(Customer.class, "c");
    private RowMapper<Email> emailMapper = BeanMapper.of(Email.class, "e");
    private RowMapper<EmailType> emailTypeMapper = BeanMapper.of(EmailType.class, "et");

    @Override
    public Customer getFinalGroup() {
        return customer;
    }

    @Override
    public Customer map(ResultSet rs, StatementContext ctx) throws SQLException {
        if (customer == null || !customer.getId().equals(rs.getInt("c_id"))) {
            returnedCustomer = customer;

            customer = customerMapper.map(rs, ctx);
        } else {
            returnedCustomer = null;
        }

        Email email = emailMapper.map(rs, ctx);
        email.setEmailType(emailTypeMapper.map(rs, ctx));
        customer.addEmail(email);

        return returnedCustomer;
    }

}

Not much has changed but now we can concat the two streams and will get all the customer objects from the resulting stream.

public Pair<Stream<Customer>, Handle> list() {

    // get the database connection
    Handle handle = jdbi.open();

    CustomerControlBreakMapper mapper = new CustomerControlBreakMapper();
    handle.registerRowMapper(mapper);

    // using SQLObjects for executing the SQL
    CustomerDao dao = handle.attach(CustomerDao.class);

    // get the stream from the ResultIterable
    Stream<Customer> customerStream = dao.list().stream();

    // get the stream with the final customer
    Stream<Customer> finalGroupStream = mapper.stream();

    // concat those two streams to one stream
    Stream<Customer> stream = Stream.concat(customerStream, finalGroupStream).filter(Objects::nonNull);

    return Pair.of(stream, handle);

}

Wrap Up

Streaming doesn’t exclude grouping if done right. And the result is pretty manageable with small classes and interfaces.

If you found another solution to the grouping / row reducing problem I would be really interested in hearing it.

Happy streaming!

Mihael

Streaming data ... the client side

Mon, 13 Jan 2025 00:00:00 +0100

Here comes the counterpart to the “Jdbi and Streaming” post. This post will cover the client side of things. We will use Jackson’s JsonParser and ObjectMapper to process the streamed data chunk by chunk … or rather JSON object by JSON object :)

This implementation doesn’t rely on any specific web framework as it just operates on an InputStream. And as it uses just an InputStream it doesn’t even necessarily has to be the response of a web server but could also be a FileInputStream for a file on disk.

Code

Most of the code is from the StackOverflow question “How to parse a JSON input stream?”.

public class JsonObjectIterator<T> implements Iterator<T>, Closeable {

    private final InputStream inputStream;
    private JsonParser jsonParser;
    private boolean isInitialized;

    private T nextObject;
    private ObjectMapper objectMapper;
    private Class<T> clazz;

    public JsonObjectIterator(final InputStream inputStream, Class<T> clazz) {
        this(inputStream, clazz, new ObjectMapper());
    }

    public JsonObjectIterator(final InputStream inputStream, Class<T> clazz, ObjectMapper objectMapper) {
        this.inputStream = inputStream;
        this.isInitialized = false;
        this.nextObject = null;
        this.objectMapper = objectMapper;
        this.clazz = clazz;
    }

    private void init() {
        this.initJsonParser();
        this.initFirstElement();
        this.isInitialized = true;
    }

    private void initJsonParser() {
        final JsonFactory jsonFactory = objectMapper.getFactory();

        try {
            this.jsonParser = jsonFactory.createParser(inputStream);
        } catch (final IOException e) {
            throw new RuntimeException("There was a problem setting up the JsonParser: " + e.getMessage(), e);
        }
    }

    private void initFirstElement() {
        try {
            // Check that the first element is the start of an array
            final JsonToken arrayStartToken = this.jsonParser.nextToken();
            if (arrayStartToken != JsonToken.START_ARRAY) {
                throw new IllegalStateException(
                        "The first element of the Json structure was expected to be a start array token, but it was: " + arrayStartToken);
            }

            // Initialize the first object
            this.initNextObject();
        } catch (final Exception e) {
            throw new RuntimeException(
                    "There was a problem initializing the first element of the Json Structure: " + e.getMessage(), e);
        }

    }

    private void initNextObject() {
        try {
            final JsonToken nextToken = this.jsonParser.nextToken();

            // Check for the end of the array which will mean we're done
            if (nextToken == JsonToken.END_ARRAY) {
                this.nextObject = null;
                return;
            }

            // Make sure the next token is the start of an object
            if (nextToken != JsonToken.START_OBJECT) {
                throw new IllegalStateException(
                        "The next token of Json structure was expected to be a start object token, but it was: "
                                + nextToken);
            }

            // Get the next object and make sure it's not null
            this.nextObject = this.jsonParser.readValueAs(clazz);
            if (this.nextObject == null) {
                throw new IllegalStateException("The next parsed object of the Json structure was null");
            }
        } catch (final Exception e) {
            throw new RuntimeException("There was a problem initializing the next Object: " + e.getMessage(), e);
        }
    }

    @Override
    public boolean hasNext() {
        if (!this.isInitialized) { this.init(); }

        return this.nextObject != null;
    }

    @Override
    public T next() {
        // This method will return the current object and initialize the next object so
        // hasNext will always have knowledge of the current state

        // Makes sure we're initialized first
        if (!this.isInitialized) { this.init(); }

        // Store the current next object for return
        final T currentNextObject = this.nextObject;

        // Initialize the next object
        this.initNextObject();

        return currentNextObject;
    }

    @Override
    public void close() throws IOException {
        try {
            this.jsonParser.close();
        } catch (Exception e) {
            // close quietly
        }

        try {
            this.inputStream.close();
        } catch (Exception e) {
            // close quietly
        }
    }

}

So you just need to get the InputStream (from the HTTP client of your choice) and pass it to the JsonObjectIterator. As we want to iterator over a number of JSON objects received from the web service the JsonObjectIterator expects a JSON array as the top level entity in the InputStream.

All in all … no magic to be found here ;-)

Wrap Up

It is pretty easy to process a big JSON array chunk by chunk with the help of the Jackson library. I skipped the part on how to use the JsonObjectIterator as it is just an iterator and I am sure you already know how to code a while loop with calling hasNext and next.

Happy streaming!

Mihael

ILE languages can also play in the streaming game

Sun, 15 Dec 2024 00:00:00 +0100

IBM i as a platform was always able to participate in the streaming game by leveraging Java or Node.js. But integrating the ILE environment in a process which needs access to a streaming platform was not as straight forward as it should be. Many streaming middlewares support access through HTTP but this is far from optimal and often not very feature rich.

But what is data streaming?

Data streaming is about processing data in (near) real-time as it arrives. It focuses on the continuous flow of data, high throughput and short processing times.

Well-known open source frameworks for data streaming are

Many cloud platform providers offer their own flavour of a commercial data streaming service.

How to join the fray?

The ILE environment has often been neglected by IBM. There is no easy and straight forward way to send/receive data to/from a streaming platform using ILE languages. Many of these platforms ship their own client library for various languages (but of course not for any ILE language or at least not usable in the ILE environment as some ship C/C++ versions but not compatible with the outdated ILE C/C++ compilers).

Is there no common nominator for a protocol to use?

In the message queueing and processing area there are some protocols which are widely used and implemented by almost every message queueing system. MQTT, AMQP, STOMP to name a few.

What about data streaming?

There are some streaming protocols like RTMP, WebRTC, FTL, HLS or SRT but these have a very specific use case (mostly video and audio streaming).

So is there really no way of streaming data with a common protocol?

If we leave the specific protocols and look at more generic solutions there is a protocol which is not an obvious candidate but should be looked at more closely: WebSocket

In a nutshell, WebSocket is a technology that enables bidirectional, full-duplex communication between client and server over a persistent, single-socket connection. The intent is to provide what is essentially an as-close-to-raw-as-possible TCP communication layer to web application developers while adding a few abstractions to eliminate certain friction that would otherwise exist concerning the way the web works. A WebSocket connection starts as an HTTP request/response handshake; beyond this handshake, WebSocket and HTTP are fundamentally different. — websocket.org

What makes this protocol interesting is the fact that it is not too specific about the use cases. And as its requirement is just a socket connection it can be used in any environment which supports sockets, including server to server use cases and not only the web/browser.

One step closer … but where is the implementation?

Here! :)

The ILEvator project supports the WebSocket protocol with its latest contribution. The user guide shows how to initiate a WebSocket connection and how to send and receive data. It has been tested with the reference implementation for the Jakarta WebSocket specification, Eclipse Tyrus.

ILEvator also works with Apache Pulsar and the Node.js WebSocket server from the ws library. Apache Kafka does not support WebSocket by itself but there are some open source tools which utilize the Node.js ws library and thus can be used with ILEvator. Also some cloud platforms support WebSocket by providing their own WebSocket service to their Kafka cluster.

Wrap Up

All it took was about 800 lines of code to let you join the streaming game. It is sad to see that there is not more effort to integrate the ILE environment in the computing environments of today.

Happy streaming!

Mihael

Jdbi and Streaming

Thu, 12 Dec 2024 00:00:00 +0100

About half a year has this post been in my draft stash. Finally I found some time and motivation to finish it. And here it is! :)

The streaming of data block by block from your web service can have significant advantages in comparision to collecting all the data on the server side and then sending it to the client.

less memory consumption
faster processing time

Less Memory Consumption

In most cases when it comes to web services the data will be collected on the server and then sent to the client. And this is totally ok and works pretty well as in most cases the data transfered from and to web services is not so big.

But when it comes to transfering several hundreds of MB in one request it may not be so efficient and may even kill the server because of consuming all the available memory … ever seen the OutOfMemoryError? ;-)

But what if you just process one entity at a time and send that chunk of data to the client? Memory consumption wouldn’t be any problem at all.

Having a solution for the server is great. But you also have to take care of the client side (if it is in your hands) so that you don’t just push the memory consumption problem from the server to the client. You need to process the data on the client also block by block.

Faster Processing Time

When streaming the data from the server to the client without having to first collect the whole data on the server side the client gets the first block of data much faster and thus can process the data earlier. Thus the whole processing time is shorter.

Jakarta JAX-RS Streaming Support

JAX-RS supports the streaming of data via the interface jakarta.ws.rs.core.StreamingOutput which can be returned as the response entity.

1	return Response.ok(myStreamingOutput).build();

Jdbi Streaming Support

Jdbi supports the streaming of data directly from an SQL query out-of-the-box in the Core API and also in the SQL Objects module with the interface org.jdbi.v3.core.result.ResultIterable. This interface can be used instead as a return type just like java.util.List.

The interface ResultIterable comes with multiple methods to work with the data. Just keep in mind that almost all these methods work similar as in java.util.stream.Stream as they are almost all terminal operations. With executing these terminal operations you will load and process all data from your query and cannot process the data again with the same ResultIterable object.

From the Jdbi docs:

Almost all operations on the ResultIterable interface are terminal operations. When they finish, they close and release all resources, especially the ResultSet and PreparedStatement objects.

DAO

The DAO interface (using Jdbi SQL Objects) just returns a ResultIterable instead of a list.

1 2	@SqlQuery("SELECT ... FROM ... WHERE ...") ResultIterable<Location> iterate();

Data Service

You can use the method stream() to get a java.util.stream.Stream object from the ResultIterable. After processing the data from the stream all the allocated resources should be freed but the Handle instance is still open. You need to pass the handle to the StreamingOutput instance so that you can call close() on the Handle instance after the processing. One way of returning both the stream and the handle is by using the Pair interface of the Apache Commons Lang project.

1	Pair.of(resultIterable.stream(), handle);

All this can be encapsulated into a service class which is then used by the web service resource class.

JAX-RS + Jdbi

Both parts (JAX-RS and Jdbi) support the streaming of data. Now we just have to bring those two together. We need to implement StreamingOutput for this and have to have access to the stream and the Jdbi Handle object.

This implementation will output a stream of objects to the client. The objects are deserialized by using an ObjectMapper instance from the Jackson Data Bind library. It also lets you limit the number of objects you want to return to the client by passing the size parameter to the constructor.

The stream and the Jdbi handle will be passed to the implementation by using the Pair interface from the Apache Commons Lang project.

The Jdbi handle will be closed after processing all the objects in the stream.

public class JdbiStreamingOutput<T> implements StreamingOutput {

    private ObjectMapper mapper;
    private Pair<Stream<T>, Handle> streamHandle;
    private Long size;

    public JdbiStreamingOutput(Pair<Stream<T>, Handle> streamHandle, ObjectMapper mapper) {
        this.streamHandle = streamHandle;
        this.mapper = mapper;
    }

    public JdbiStreamingOutput(Pair<Stream<T>, Handle> streamHandle, ObjectMapper mapper, Long size) {
        this.streamHandle = streamHandle;
        this.mapper = mapper;
        this.size = (size != null && size < 0) ? 0 : size;
    }

    @Override
    public void write(OutputStream out) throws IOException, WebApplicationException {
        AtomicInteger i = new AtomicInteger();
        byte[] comma = ",".getBytes(StandardCharsets.UTF_8);

        out.write("[".getBytes(StandardCharsets.UTF_8));

        try {
            Stream<T> stream = streamHandle.getLeft().sequential();
            if (size != null) stream = stream.limit(size);

            stream.forEach(c -> {
                try {
                    if (i.get() > 0) out.write(comma);
                    mapper.writeValue(out, c);

                    i.incrementAndGet();
                } catch (Exception e) {
                    throw new RuntimeException("Object deserialization failed.", e);
                }
            });
        } catch (Exception e) {
            throw new RuntimeException("Error on streaming data at element " + (i.get()+1), e);
        } finally {
            if (!streamHandle.getRight().isClosed()) streamHandle.getRight().close();
        }

        out.write("]".getBytes(StandardCharsets.UTF_8));

        out.flush();
    }
}

Web Service Resource

In the web service resource class a JdbiStreamingOutput instance is returned as the resposne entity … and that’s it :).

public Response stream() {
    Pair<Stream<Location>, Handle> result = locationService.stream();
    return Response.ok(new JdbiStreamingOutput<Location>(result, mapper)).build();
}

An instance of ObjectMapper is passed for deserialization to the constructor.

Final Thoughts and Hints

Of course there are some limitation to this simple implementation. F. e. you can only return an array of objects (which is mostly the case when you stream JSON data ;-) ).

It is also tied to the libraries Jdbi and Jackson Data Bind. But this is intended and pretty easy to abstract away if needed.

Hint

@UseRowReducer cannot be used in this case in the Jdbi DAO interface as it results in Jdbi collecting all the data on the server before streaming it and thus … no streaming. But there is a solution which will be presented in another post.

Wrap Up

This post demonstrates how to stream data from the database to the client using JAX-RS and Jdbi. There will be a follow up post about the other side … the client side.

Happy streaming!

Mihael

RPG, OpenID Connect and M2M ... or WTF?!

Thu, 01 Aug 2024 00:00:00 +0200

Just listing some buzz or tech words in a title is not one of the best ways to convey an information to an audience. The target audience might not even realize that the article could be of interest to them because the title is so hard to grasp and understand. None the less … here we are :) .

For some RPG developers their decades old unchanged environment is slowly crumbling and some tasks require to integrate some new stuff … like web services, access tokens and JWT.

Some developers will happily yell “Hooray!” while others rather reluctantly engage this challenge.

One such challenge might be sending some data (f. e. customer master data) to some service provider.

One de facto industry standard for exchanging data is the usage of web services and access tokens (for authentication and authorization).

Many service providers use the OpenID Connect standard, parts of it or at least very similar concepts.

OpenID Connect

OpenID Connect (OIDC) is an authentication protocol built on OAuth 2.0 that allows applications to verify the identity of users and obtain basic profile information. It provides an identity layer on top of OAuth 2.0, using JSON Web Tokens (JWT) for securely transmitting user information. OpenID Connect provides several authentication flows to cover most use cases.

In our case (sending customer master data to a service provider via a web service) we have two programs (machines) which will communicate with each other. There is no human interaction. This is also known as M2M (machine to machine).

One part of securing the communication is by sending an access token alongside the actual data. The access token can be requested from an IAM service (Identity and Access Management). Most IAM services support the OpenID Connect protocol.

OIDC supports the M2M use case with the Client Credentials flow. This is a very simple flow where you pass the credentials form urlencoded to the IAM service and get an access token in return.

How to do that in RPG?

You can do that in RPG in several ways. IBM i and the SQL services provide everything you may need for this challenge but it is not always as straightforward as I would like things to be. The tools I would choose are ILEvator (HTTP client) and noxDB (JSON parser and builder).

First you need to install both packages. There are several documentations showing how to do this.

In your code add the copy books.

1 2	/include 'ilevator.rpgle' /include 'jsonxml.rpgle'

The code for getting the access token is encapsulated in its own procedure.

dcl-proc getAccessToken;
    dcl-pi *n varchar(10000);
        url varchar(10000) const;
        clientId varchar(100) const;
        clientSecret varchar(100) const;
        scope varchar(1000) const options(*nopass);
    end-pi;

    dcl-s ilevator pointer;
    dcl-s token varchar(10000);
    dcl-s headers pointer;
    dcl-ds formData likeds(iv_lvarpuchar_t);
    dcl-s responseData varchar(32766);
    dcl-s json pointer;
   
    ilevator = iv_newHttpClient();

    // We need to register a certificate store if we want to access resources via HTTPS.
    // A default certificate store can be obtained from the ILEvator project repository.
    iv_setCertificate(ilevator : '/usr/local/etc/certs/ilevator.kdb' : 'ilevator');
   
    if (%parms() >= 4);
        formData = iv_form_of(
            'grant_type' : 'client_credentials' :
            'client_id' : clientId :
            'client_secret' : clientSecret :
            'scope' : scope
        );
    else;
        formData = iv_form_of(
            'grant_type' : 'client_credentials' :
            'client_id' : clientId :
            'client_secret' : clientSecret
        );
    endif;
   
    headers = iv_buildList(
        'Accept' : 'application/json' :
        'Content-Type' : 'application/x-www-form-urlencoded'
    );

    responseData = iv_postForm(url : formData : headers);

    json = jx_parseString(responseData);

    // The returned JSON should at least have an attribute "access_token" 
    // according to the OpenID Connect spec which contains the token.
    token = jx_getStr(json : 'access_token');

    return token;

    on-exit;
        iv_form_free(formData);
        iv_freeList(headers);
        iv_free(ilevator);
end-proc;

Is this good code ready to use in production? I think not. But that was not my intention in the first place. This should just give you some starting points on your journey into the World Wild Web ;-) .

Access Token Expiration

If you just want to do a one-shot (getting an access token and using it in just a single web service call) you don’t have to consider anything regarding the lifetime of the access token. If you want or need to use the token in several calls you may find yourself in the situation where the token has expired (which normally results in a failed web service call with an HTTP status 401 from the called web service).

“expired” … what does that mean?

A JWT access token is generated at a given moment. This moment is saved in the token with the claim (attribute) iat (issued at). The value for this claim is a unix timestamp (which are the seconds counted from the 1970-01-01 at 00:00 o’clock UTC).

If nothing else is stated this token is valid for any moment in the future. But most tokens have an explicit lifetime. This lifetime is expressed in another claim : exp (expiration). The value of this claim is a unix timestamp at which this token will expire and is no longer valid.

The ILEastic project provides a JWT service program which can be used to check if a token has expired, see service program JWT procedure jwt_isExpired.

Usage of an Access Token

Most of the time you will need to pass the access token alongside the data. When used with the HTTP protocol you will need to pass the access token as a bearer token in the Authorization header entry.

This can be achieved very easily in ILEvator.

dcl-s headers pointer;

headers = iv_buildList(
    'authentication' : 'Bearer ' + accessToken
);

string = iv_get('http://localhost' : IV_MEDIA_TYPE_JSON : headers);

iv_freeList(headers);

Note that the keyword Bearer is case sensitive though some software libraries are more lenient than others and will also accept other lower-/uppercase combinations.

The memory for the header list is allocated dynamically. So you always need to call iv_freeList to free any used memory of the list.

Wrap Up

If you got the basics figured out and know which tools to use it is pretty simple to get an access token and use it in your application. There is no magic to it.

Happy web servicing!

Mihael

ILEastic - JSON Web Key

Sat, 22 Jun 2024 00:00:00 +0200

Securing you web services at a single/central point like an API gateway is really a nice thing. But as long as someone can access the web service without going through that single point the web service should also do some authentication.

Welcome to JWT and JWK!

JWT provides an essential part of a standardized way for authenticating a request. JWK is rather optional but makes it much easier how secrets and keys are provided to the verifier of token signatures.

I won’t go into too much detail about JWT and JWK here. You can find many web sites explaining both, JWT and JWK.

JSON Web Key

The JSON Web Key describes the necessary details on how to verify or encrypt when it comes to JWT. And as the name suggests the data is provided as a JSON object.

Attributes:

kid - key id : Identifies the web key
kty - key type : What type of key is stored in this web key
alg - algorithm : Algorithm to be used with this key
use : What this key should be used for
key_ops : What key operations the provided key is intended to be used with (similar to “use”)
oct : key , base64url encoded
x5c : array of public keys, binary (DER) format base64 encoded (not base64url)

The mentioned attributes are not a full list but rather what is supported by this JWK implementation. No all of these attributes are required for every web key.

Example for HS256:

{
    "kid": "123-456-789",
    "kty": "oct",
    "alg": "HS256",
    "use": "sig",
    "k": "MTIzNDU2Nzg5MDEyMzQ1Njc4OTAxMjM0NTY3ODkwQUI="
}

Example for RS256:

{
    "kid": "123-456-789-0",
    "kty": "RSA",
    "alg": "RS256",
    "use": "sig",
    "x5c": [
    "MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA0CfTwHgz2CoNTxx8KYVmstdsVJrvgKqyiui7fO+VFga1pAAPcUsNtU3aRFRE9FWLId9z0AoEdYV7pOWxfFrqJ/M63MByIFfnVLGdR7k6MKKFaiTPNAv7/qAiWvIDHPIyCCbfRRuJ44LOUFTyn6fp9cnUNpAxNCTLJvhebaPB4zF/QroetOHizGfoRybBJyFZ2Rmuog/KKmdgBQSaPqTksOg+SOjf0BuGvycIaIr3rLeK0WO4o9GRBQCtLK6sURwNNO4F7LBPHVC1rtX0GnEZ7FmpOu0UopvYevxgwwNb2FQ5XY3mW4tiAMZLphJehjizXqcghYwu8sczb7QEz611KQIDAQAB"
    ]
}

JSON Web Key Set

Multiple JSON Web Keys can be added to a JSON Web Key Set (JWKS) which is just a JSON object with the attribute keys. The value of keys is a simple JSON array where each JSON Web Key is an element in that array.

{
  "keys": [
    {
      "kid": "123-456-789",
      "kty": "oct",
      "alg": "HS256",
      "use": "sig",
      "k": "MTIzNDU2Nzg5MDEyMzQ1Njc4OTAxMjM0NTY3ODkwQUI="
    },
    {
      "kid": "456-789-abc",
      "kty": "oct",
      "alg": "HS512",
      "use": "sig",
      "k": "MTIzNDU2Nzg5MDEyMzQ1Njc4OTAxMjM0NTY3ODkwQUIxMjM0NTY3ODkwMTIzNDU2Nzg5MDEyMzQ1Njc4OTBBQg=="
    }
  ]
}

Implementation

The JWK implementation in ILEastic supports the key types oct and RSA. oct is used for HMAC-SHA algorithm and RSA … is self-explanatory :-) .

The key type oct requires also the attribute oct. When using the key type RSA the implementation expects the public key to be base64 encoded (not in PEM format) in the attribute x5c.

For token verification either the attribute use with the value sig or key_ops with the value verify needs to be present.

Loading a JWK

If used with ILEastic JWK and JWKS should be saved in a stream file in the IFS. The content of the stream file is loaded as is. No character conversion takes places. The content needs to be in an ASCII compatible format like UTF-8 (CCSID 1208) or LATIN1 (CCSID 1252).

For loading the JWK we need to add at least the JWT and JWT plugin copy books.

1 2	/include 'jwt.rpginc' /include 'jwtplugin.rpginc'

The procedure il_jwt_addVerifyOptionsFromJwks loads a JWK and JWKS from the stream file and adds it to the ILEastic JWT configuration.

Note: By default when loading a JWK with il_jwt_addVerifyOptionsFromJwks the verify options lookup method will be “by kid”. That means the JWT plugin will choose a JWK which matches the kid attribute in the JWT from the client request.

Selecting a Verify Option

JWKs are added to the plugin configuration as verify options. When adding a verify option an id can be passed for this verify option. This id will be used for determining which verify option to use by comparing it with an attribute from the JWT token.

The ILEastic JWT plugin must decide which JWK to use on a client request. The plugin offers three options:

JWT_VERIFY_OPTIONS_LOOKUP_NONE : The first registered verify option will be used on every request.
JWT_VERIFY_OPTIONS_LOOKUP_ISSUER : The issuer claim in the JWT token payload will be used for selecting a matching verify option.
JWT_VERIFY_OPTIONS_LOOKUP_KID : The kid attribute of the JWT header will be compared with the kid attribute of the JWKs. The JWK of the matching one will be used for token verification.

When adding verify options by calling il_jwt_addVerifyOptionsFromJwks the kid will be automatically used as the id.

Note: JWT_VERIFY_OPTIONS_LOOKUP_ISSUER cannot be used in combination with il_jwt_addVerifyOptionsFromJwks as the comparing of kid and iss doesn’t make sense.

Wrap Up

Industry standards like JWT and JWK have found their way into ILEastic and are well supported. If your case is not supported just create an issue on GitHub and present your case.

Happy web serving!

Mihael

Helidon + HikariCP + JT400 = IllegalArgumentException

Sat, 08 Jun 2024 00:00:00 +0200

Helidon has extra support for using HikariCP. A DataSource instance can be easily created with MicroProfile Config and CDI.

<dependency>
    <groupId>io.helidon.integrations.cdi</groupId>
    <artifactId>helidon-integrations-cdi-datasource-hikaricp</artifactId>
</dependency>

The Problem

Helidon adds HikariCP 5.0.1 as a dependency. This version of HikariCP supports setting the password as a String on the DataSource. But String as a variable type for a password is more insecure than a primitive data type like a char array, see StackOverflow question Why is char[] preferred over String for passwords?. Some drivers added support for char array for setting the password and overloaded the method setPassword, f. e. the JTOpen project with the JDBC driver for DB2 on i. In case HikariCP gets the setPassword method with the char array as a parameter that will result in an IllegalArgumentException as it tries to pass the password as a string to it.

Failed to set property password on target class com.ibm.as400.access.AS400JDBCDataSource
    java.lang.IllegalArgumentException: argument type mismatch
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.base/java.lang.reflect.Method.invoke(Unknown Source)
        at com.zaxxer.hikari.util.PropertyElf.setProperty(PropertyElf.java:154)
        ...

So depending on what of version of setPassword the Java VM returns first it will result in an IllegalArgumentException or not.

The Solution

HikariCP version 5.1.0 supports String and char array as a parameter for setPassword. To get Maven choose version 5.1.0 over the managed version of the Helidon BOM artifact you need to add the HikariCP version 5.1.0 to your POM.

Now the version 5.1.0 of HikariCP should be nearer to the root of the dependency tree than the version managed by Helidon and thus should be selected by Maven.

In Eclipse you can check this by open the pom.xml in the POM editor and open the tab Dependency Hierarchy. It displays the resolved dependencies and their versions.

<dependency>
    <groupId>com.zaxxer</groupId>
    <artifactId>HikariCP</artifactId>
    <version>5.1.0</version>
</dependency>

Happy connecting!

Mihael

Tour of Champions - OpenAPI

Sun, 26 May 2024 00:00:00 +0200

What good is a tool if you don’t know how to use it?

… and the same goes for web services. Without a good documentation on how to use the provided web service the value of that web service is lowered massively. In the worst case it is not usable at all.

But instead of writing pages of unstructured prose about the web service the well established standard OpenAPI should be used as it is very well known in the web development world and has lots of tooling support. OpenAPI originated at Swagger which offers extra tooling for OpenAPI, f. e. an online OpenAPI editor and the Swagger UI project.

The most common format for providing an OpenAPI document is YAML and JSON.

But hey, YAML and JSON are data exchange formats. I thought we should write a documentation?!?

Yes. But 99% of web services today are using HTTP and HTTP has a well defined structure and syntax. So for many parts of the documentation you just have to enter some keywords. The rest will be generated.

Hint: You can see how your OpenAPI document looks like in web UI with the Swagger OpenAPI editor. It also shows you any syntax errors you might have in your document.

Where to start?

More or less the OpenAPI document has 3 main parts: common description, paths and components (data model).

Depending on the tooling you might want to take an existing document and edit that one. The petstore example is a good place to start.

Ways to Integrate

You have your OpenAPI document finished. Now what?!

Again … depending on the tooling you use you may want to provide the OpenAPI document via the web service framework (ILEastic in our case). An easy way to do it is to write your own /openapi endpoint / route procedure and simply call il_servceStatic. This will write the content of a stream file (your OpenAPI document residing in the IFS) to the stream returned to the client (f. e. Swagger UI).

That sounds good but now you need to deploy the program object to the target server and the stream file. That is doable and most change management systems support that but it is more complicated as it needs to be.

ILEastic OpenAPI static plugin to the rescue!

ILEastic provides a plugin for generating and serving the OpenAPI document, openapi-static.

The tooling provided by this plugin lets you generate an ILE module from the OpenAPI document. This module has no functions or procedures. It just contains the content of the OpenAPI document.

Now just add the generated ILE module and two modules from the openapi-static plugin to your ILEastic web service program object and you are done. Everything is now contained inside your program object and can be easily deployed to any IBM i server just by copying the program object.

Embedded OpenAPI

Another common way is to embedd OpenAPI snippets into your program code as comments. By doing this the documentation resides alongside your code and is also managed by any version control system used.

//
// @openApiPath
//
//  /champion:
//    get:
//      tags:
//        - champion
//      summary: List IBM champions
//      description: Returns the details of all awarded IBM champions
//

The openapi-static plugin provides the command OAPISRCGEN to parse one or more source stream files on the IFS and extract the OpenAPI snippets from the source comments to a stream file on the IFS. From there on it is the same procedure as previously by creating an ILE module from the extracted OpenAPI documentation and adding it to the program modules of the program object.

This whole process is described in the README of the openapi-static plugin.

Hosting your own UI

Now you got your OpenAPI document(s) served by ILEastic but what now? A very convenient way of working with your OpenAPIs is to setup your own Swagger UI instance and configure your various OpenAPI documents there. This can be easily done with an Apache HTTP server instance on IBM i.

Wrap Up

ILEastic is a very flexible and extendable framework which can cover every needs the modern developer may have. If a topic is not covered yet it is often not very hard to implement it yourself.

Happy OpenAPI serving!

Mihael

Tour of Champions - JWT

Sun, 19 May 2024 00:00:00 +0200

For a long time Basic Auth was the defacto standard in authentication when it comes to HTTP and the web. But those times are fast fading into oblivion as standards like OAuth and JWT have established themselves very well.

ILEastic has supported Basic Auth and JWT authentication for some time now. Recently the JWT plugin has gotten a contribution which has now been adapted to the rest of the ILEastic code base. The adaption resulted in an incompatible API to the last version but added support for RSA asymmetric algorithm.

The new JWT service program version supports the following algorithms for verification:

HS256
HS384
HS512
RS256
RS384
RS512

Example

**FREE

ctl-opt dftactgrp(*no) actgrp(*caller) main(main) bnddir('ILEASTIC');


/include 'ileastic.rpgle'
/include 'jwt.rpginc'
/include 'jwtplugin.rpginc'


dcl-proc main;
  dcl-ds config likeds(IL_CONFIG);
  dcl-ds jwtOptions likeds(jwt_options_t) inz;

  config.port = 35801;
  config.host = '*ANY';

  jwtOptions.key = 'my_really_secret_key_which_must_be_at_least_32_chars_long';
  il_jwt_addVerifyOptions(jwtOptions);
  il_addPlugin(config : %paddr('il_jwt_filter') : IL_PREREQUEST);

  il_addRoute(config : %paddr(sayHello) : IL_GET : '/hello');

  il_listen (config);
end-proc;


dcl-proc sayHello;
  dcl-pi *n;
    request  likeds(IL_REQUEST);
    response likeds(IL_RESPONSE);
  end-pi;

  dcl-s name varchar(100);

  name = il_getQueryParameter(request : 'name' : 'World');

  response.contentType = 'text/plain';
  il_responseWrite(response : 'Hello ' + name);
end-proc;

Load Sign Key from Stream File

In the example above the key for verifying the JWT token signature is directly set in the code. Normally you wouldn’t do this. The key would be loaded from some external source like a file so that it could be changed without the need to change and recompile the whole program. The JWT plugin directly supports loading the key from a stream file like this:

1	jwtOptions.key = il_jwt_loadKeyFromStreamFile('/path/to/signkey/file');

The content of the file will be read in binary mode. There will be no character conversion. If it is a sign key for the HS algorithm or a RSA public key (PEM format) the content should be in an ASCII compatible character set like 1208 (UTF-8) or 1252 (Latin1).

JWT Algorithm “none”

Besides the listed supported algorithms for JWT tokens it is also worthy to mention that the JWT algorithm “none” will always be rejected as it is considered insecure.

Specific JWT Algorithm

To make things even more secure you can optionally specify a JWT algorithm in the options. JWT tokens with a different algorithm will be rejected.

1	jwtOptions.alg = JWT_RS256;

Multi-Tenant Support

Previously only a single key could be configured for JWT verification. Now multiple keys can be added to the JWT plugin configuration by adding an id on registration.

jwtOptions.key = il_jwt_loadKeyFromStreamFile('/var/local/etc/rpgenxtgen.pub');
il_jwt_addVerifyOptions(jwtOptions : 'https://rpgnextgen.com');

jwtOptions.key = il_jwt_loadKeyFromStreamFile('/var/local/etc/sitemule.pub');
il_jwt_addVerifyOptions(jwtOptions : 'https://sitemule.com');

But how will ILEastic decide which key to use for JWT verification? ILEastic supports the lookup of the key by the “kid” attribute in the JWT header and by the issuer claim (iss) in the payload. The value of the id must match with the value of the attribute from the JWT token.

The method of verify options lookup can be set with the following code:

1	il_jwt_setVerifyOptionsLookupMethod(JWT_VERIFY_OPTIONS_LOOKUP_ISSUER);

JWT payload:

{
    "sub" : "mihael@rpgnextgen.com",
    "iss" : "https://rpgnextgen.com",
}

The above payload of a JWT token would match the first configuration because the issuer claim matches the id (second parameter) on the il_jwt_addVerifyOptions call.

Wrap Up

Besides some changes in the length of some template variables the JWT plugin has gotten some major update. Thanks for the contributions.

Happy web serving!

Mihael

Tour of Champions - Load Balancing

Sun, 21 Jan 2024 00:00:00 +0100

From time to time you may enouter the situation where you have a very well written web service which works great for 95% of time but you have some spikes where there is more traffic on those services as usual and much more than one instance can handle.

One approach is to look for some ways to optimize things so it could also handle those spikes. If an optimization results in being able to handle the traffic spikes it would be the ideal situation as this would keep things simple and “simple” is something I really like ;-).

But sometimes there is not much we can do about the performance of a single web service instance any more. But if one instance cannot handle things then maybe some more instances can.

Welcome to the land of load balancing!

Load Balancing

Here some definition for load balancing from Wikipedia (yeah … shame on me for using Wikipedia as a reference ;-) ):

In computing, load balancing refers to the process of distributing a set of tasks over a set of resources (computing units), with the aim of making their overall processing more efficient.

So we have a set of tasks (multiple HTTP requests) which we would like to distribute over a set of resources (multiple instances (jobs) of a web service).

Starting the web service program in multiple jobs (on a different port for each job) is really no problem on our beloved IBM i. There are so many ways to skin this cat like SBMJOB or using autostart jobs in a subsystem or … . Choose the variant which best fits your environment.

So now we have the set of resources set up and we can easily generate our set of tasks (HTTP requests) with tools like k6 or Gatling. The missing piece of the puzzle is the middleware for the load balancing itself.

But you don’t have to look any further. One possible solution is sitting right in front of your nose. It is included in every IBM i installation and is installed on millions of servers.

Apache HTTP Server!

Apache HTTP Server

The Apache HTTP server project support its features by including and enabling modules. All modules of the Apache HTTP server has been compiled as ILE modules and some of them are bound to the service program QHTTPSVR/QZSRCORE. At least this service program contains the HTTP server modules which are needed for setting up load balancing for the HTTP server.

The necessary modules are

mod_proxy
mod_proxy_http
mod_proxy_balancer
load balancer scheduler algorithm module like mod_lbmethod_byrequests

The modules will be loaded with the LoadModule directive. The syntax is very simple:

1	LoadModule <module identifier> <file>

The module identifier of the module can be found in the Apache HTTP documentation for modules.

Most tutorials and documentation for the Apache HTTP server is for the Linux operating system. The corresponding object of a service program in linux is a shared object (suffix .so). So if you see a linux tutorial with this line for loading a module

1	LoadModule proxy_module modules/mod_proxy.so

it corresponds to this line on IBM i:

1	LoadModule proxy_module /QSYS.LIB/QHTTPSVR.LIB/QZSRCORE.SRVPGM

So loading the modules necessary for load balancing looks like this:

LoadModule proxy_module /QSYS.LIB/QHTTPSVR.LIB/QZSRCORE.SRVPGM
LoadModule proxy_http_module /QSYS.LIB/QHTTPSVR.LIB/QZSRCORE.SRVPGM
LoadModule proxy_balancer_module /QSYS.LIB/QHTTPSVR.LIB/QZSRCORE.SRVPGM
LoadModule lbmethod_byrequests_module /QSYS.LIB/QHTTPSVR.LIB/QZSRCORE.SRVPGM

The rest of the configuration is compatible with the standard documentation and may look like this:

ProxyRequests Off
<Proxy \*>
    Order deny,allow
    Deny from all
</Proxy>

<Proxy balancer://clusterIleastic>
    BalancerMember http://devil01:47001
    BalancerMember http://devil01:47002
    ProxySet lbmethod=byrequests
</Proxy>

<Location /balancer-manager>
    SetHandler balancer-manager
</Location>

ProxyPass /balancer-manager !
ProxyPass / balancer://clusterIleastic/

Note: I am no expert of Apache HTTP server. This is a simple configuration for my private adventures and should definitely be reviewed before using it anywhere.

ILEastic Web Services

As we can see “Load Balancing” is not something specific about ILEastic. It is rather the opposite: it has nothing directly to do with ILEastic. But ILEastic web services can be “load balanced” in a very easy way because an ILEastic web service brings everything to the table it needs … even its own web server. So all you really need to do is keep the port number configurable and you are ready to go.

Starting a new instance of an ILEastic web service is a simple SBMJOB and stopping an instance is done with ENDJOB. No magic there.

It really can be that simple!

Happy load balancing!

Mihael

i means integration - Receiving data with Helidon

Mon, 08 Jan 2024 00:00:00 +0100

When the IBM i community talks about programming languages on the IBM i it is mostly about RPG and CL. Sometimes C also trickles into the conversion but mostly that’s about it. Most people would not give Java a second thought on the IBM i even though many IBM products are based on Java and are running very well on the i.

Java frameworks and libraries are very mature. And with organizations like Eclipse and Apache you get really high quality software.

What may hinder the usage of Java on IBM i is the release cycle of the Java VM and runtime from IBM. IBM releases “only” Long Term Support (LTS) releases of Java for IBM i. We had Java 11 on IBM i and it took a long time till IBM released Java 17. This really hinders adoption as the major players are dashing forward with ever shorter release cycles and more current versions of dependencies in their requirements.

The Helidon microservice framework is a good match when it comes to releases and requirements as you can use Helidon version 2 which is supported by Oracle and run it with Java 11 on older IBM i versions. But IBM has released Java 17 for IBM i some time ago so that Helidon version 3 can now also run on IBM i which comes with some improvements concerning messaging. With Helidon 3 Oracle jumped from MicroProfile Messaging 1.0 to MicroProfile Messaging 3.0.

And MicroProfile Messaging is the key component of this post as it makes it very easy to receive data from message queuing systems like ActiveMQ (Classic) or Artemis.

Could this have been possible earlier with Java on IBM i?

Yes, definitely! But with the advent of microservices and the corresponding specs in Java implementations and projects in general it is now much easier to do it.

For the examples in this post I will use the Helidon framework version 3.

Receiving Data

Originally the RPG program had sent the data as one big blob of bytes. And for as long as we stay in our RPG bubble this would have worked. Would it also work for Java? Yes, but it would take some extra steps. With the JTOpen library we could define a record format (see RFML, Record class, RecordFormat class, RecordFormatDocument class) and feed the data to that Record or RecordFormatDocument instance. Then we could extract the single pieces of data we need from that instance.

Does not sound very straight forward, doesn’t it?!

But as we also have the choice to send the data in JSON format we can simplify things a lot on the Java side because then the framework(s) is doing all the heavy lifting for us. For getting the data for an auction we can just code the following.

@Inject
ObjectMapper mapper;

@Incoming("auction-events")
@Acknowledgment(Acknowledgment.Strategy.MANUAL)
public CompletionStage<Void> receiveMessage(JmsMessage<String> message) {
    try {
        AuctionEvent event = mapper.readValue(message.getPayload(), AuctionEvent.class);
        handleEvent(event);
        return message.ack();
    } catch (Exception e) {
        logger.log(Level.SEVERE, "Error on processing auction event", e);
        return message.nack(e);
    }
}

That is pretty straight forward :).

What makes the code also very short is the fact that we can get an Auction object with a one-liner by using the Jackson library to deserialize the JSON data. Very nice library!

Note: I decided to manually acknowledge the messages as this allows us to “not acknowledge” a message which then stays in the queue and won’t be lost. This is not the ideal solution as now the message will block any further processing. You need to find a strategy which best fits your use case.

Artemis

The above solution for receiving data would be ideal. But as we all know we don’t always have the ideal situation in the real world. Mostly it looks a little bit more complicated and this is also the case here.

If we send the data with the STOMP client for ILE it will always send a Content-Length header. Artemis will always interpret the message as a ByteMessage when using JMS because of this header, see Artemis - STOMP and JMS interoperability.

But with the MicroProfile Spec we are super flexible when it comes to how to receive message from a queue.

To circumvent our problem we can inspect what type of message we got and then act accordingly.

@Inject
private ObjectMapper mapper;

@Incoming("auction-events")
@Acknowledgment(Acknowledgment.Strategy.MANUAL)
public CompletionStage<Void> receiveMessage(JmsMessage<?> message) {
    String messageText = null;

    try {
        // content-length header available -> ByteMessage else TextMessage
        if (message instanceof JmsTextMessage) {
            messageText = ((JmsTextMessage) message).getPayload();
        } else if (message instanceof JmsBytesMessage) {
            messageText = new String((byte[]) message.getPayload(), StandardCharsets.UTF_8);
        } else {
            throw new RuntimeException("Unknown JMS message type " + message.getClass().getName());
        }

        AuctionEvent event = mapper.readValue(messageText, AuctionEvent.class);
        handleEvent(event);
        return message.ack();
    } catch (Exception e) {
        logger.log(Level.SEVERE, "Error on processing auction event", e);
        return message.nack(e);
    }
}

In our example we use the well established library Jackson to convert the data from JSON to an auction object.

Logging

If you want to integrate the logs of your Java application into any central logging server then one way to achieve this is to send you logs directly to the logging server via a socket connection. The Java Util Logging framework supports this by using the SocketHandler class.

Note: You may want to reimplement the SocketHandler so that it reconnects in case it lost the connection to the logging server.

Formatter

The SocketHandler class will by default send every log entry to the log server in XML format. If you need to transfer the log entries in JSON this can be done very easily by implementing the java.util.logging.Formatter interface.

To create the JSON string you can use the ObjectMapper class from the Jackson library. To get a root node you can call ObjectMapper::createObjectNode. For adding values to the root node you can call the corresponding put method from the ObjectMapper class.

Custom Data

By default only the data from the log entry is sent to the log server. You may want to transfer additional data to the log server, f. e. the application name so that you know to which application the log entries belong to.

For static data this can be added to the configuration of the formatter.

1	rpgnextgen.log.JsonFormatter.application = rpgnextgen.helidon.mq

The formatter can query the LogManager for any logging configuration data.

1	String application = logManager.getProperty("rpgnextgen.log.JsonFormatter.application");

And later add it to the log entry.

1 2	ObjectNode root = mapper.createObjectNode(); root.put("app", application);

Dependencies

With most other programming languages it takes a lot of effort to choose the libraries you want to use. Best negative example is the world of JavaScript/TypeScript.

In our Java project using Helidon we have almost solely dependencies from Apache, Eclipse, JBoss and Oracle. Every one of them is offering high quality software packages. The remaining dependencies can be easily looked up (like the Netty project or FasterXML). This makes it much easier to check dependencies and maintain the application.

Happy integrating!

Mihael

i means integration - Receiving data with STOMP client

Sun, 10 Dec 2023 00:00:00 +0100

In the last post we have sent a message to a message queue on the Artemis server. We did this by utilizing the STOMP client service program available at repo.rpgnextgen.com.

In this part of the blog series we want to receive the message we sent in our last episode.

The message was sent with the content type application/octet-stream (constant STOMP_MIME_BINARY) which describes that we sent a bunch of binary data. The data is encoded with the CCSID of the job.

This works fine as long as we are staying on the same server and/or have full control over our environment. But it may not work so good if different parties need to communicate with each other.

So how does the code look like for receiving data?

1   **FREE
2
3   ctl-opt main(main);
4
5   /include 'stomp.rpginc'
6
7  dcl-proc main;
8      dcl-s client pointer;
9      dcl-s frame pointer;
10     dcl-ds frameBuffer likeds(stomp_util_buffer_t);
11
12     client = stomp_create('127.0.0.1' : 40002);
13     stomp_setClientId(client : 'mihael');
14
15     stomp_open(client);
16     stomp_command_connect(client : 'admin' : 'udontknow');
17     stomp_command_subscribe(client : 'auction-events');
18
19     // start receiving data
20     frame = stomp_receiveFrame(client);
21     dow (frame <> *null);
22         if (stomp_frame_getCommand(frame) = 'ERROR');
23             dsply 'An error occured';
24             leave;
25         endif;
26
27         // display message content
28         frameBuffer = stomp_frame_getBody(frame);
29         message_info(%str(frameBuffer.data : frameBuffer.length));
30         stomp_frame_finalize(frame);
31
32         // receive next frame
33         frame = stomp_receiveFrame(client);
34     enddo;
35
36     stomp_command_disconnect(client);
37     stomp_close(client);
38
39     on-exit;
40         stomp_finalize(client);
41 end-proc;

Lets have a look at the code line by line. We start exactly the same as the sending programing.

Line 1 : We are writing fully free RPG code. Not half-assing this already from the start ;-)
Line 3 : We will compile this program in two steps. Thus we only declare the main procedure in the control options.
Line 5 : The STOMP copybook needs to be included so that we can use the constants and prototypes from the STOMP project. Everything else will be added to the compile commands.
Line 8 : For the client a pointer variable is declared. We are not interested what lies behind that pointer. We just pass it to the STOMP procedures.
Line 9 : A variable for the received frames is declared.
Line 10 : The frame body is accessed via a pointer. The frameBuffer data structure contains this pointer (data) and the length of the data which lies behind the data pointer.
Line 12 : An instance of the STOMP client is created. The client is accessed via the pointer. The internal data of the client doesn’t matter to us. We are only using the procedures from the STOMP project anyway. The host and port we want to connect to are passed to the create procedure. The port should be the same we used in the Artemis configuration for our STOMP acceptor.
Line 13 : A client id is set on the client. This id can be used to identify this specific client connection to the server in the Artemis web console. The id is just a string and can be set to any value.
Line 15 : A socket connection to the Artemis server instance is created.
Line 16 : To connect on a (STOMP) protocol level we need to send a STOMP CONNECT frame/command to the server. This also includes the credentials used to login to the message broker. For an anonymous connect we can just pass the client pointer.
Line 17 : We need to tell the Artemis instance from which queue we want to receive messages. This is done by subscribing to a queue or topic (or in Artemis lingo: address).
Line 20 - 21 : Here we are starting the loop for receiving messages (STOMP lingo: frames) from the Artemis instance. The program will wait here till the next message has been received. stomp_receiveFrame will return *NULL if the timeout has been reached before any frame has been received. We can change the timeout on the client by calling stomp_setTimeout.
Line 22 : The server may also send an ERROR frame to indicate that something went wrong. We need to be ready for that and react accordingly.
Line 28 : We are mostly interested in the body of the message. We get that with stomp_frame_getBody. The returned data structure contains a pointer to the message body and a subfield for the length. We could also get the headers of the frame with stomp_frame_listHeaders and stomp_frame_getHeader if we needed to.
Line 29 : We are expecting some string data in the same CCSID as our job. Because of that we can use the %str built-in function which returns a string based on the passed pointer.
Line 30 : Evey stomp frame we receive has its memory dynamically allocated. We need to free this memory by calling stomp_frame_finalize.
Line 36 : To properly disconnect from the Artemis server instance after we have finished the loop we need to send a DISCONNECT frame to the server.
Line 37 : After disconnecting we can close the socket connection.
Line 40 : At the end we also need to free the memory allocated by the client.

We will see the content of the message in the job log (output by the message_info procedure from the MESSAGE service program).

Durable Subscriber

On subscribing to a message queue Artemis will create a queue for our client which receives all the messages for which we have subscribed. For as long as our program runs we will receive those messages. But Artemis will delete this queue (not the address) when we disconnect. This means that we might miss some messages. STOMP has no concept of a persistent subscriptions. But Artemis supports Durable Subscriptions when using the STOMP protocol. From the Artemis docs:

To create a durable subscription the client-id header must be set on the CONNECT frame and the durable-subscription-name must be set on the SUBSCRIBE frame. The combination of these two headers will form the identity of the durable subscription.

The STOMP service program has dedicated procedure for setting those values.

dcl-s client pointer;

client = stomp_create('127.0.0.1' : 40002);
stomp_setClientId(client : 'mihael');
stomp_setDurableSubscriberName(client : 'auctions');

Now we won’t miss any messages.

Content Type

Currently the message itself doesn’t convey anything about the message itself. The content type is just application/octet-stream which means that it can be anything. It is just a bunch of bytes. The receiving program needs to know a lot about the sending program and thus has a tight coupling to it. And as long as we just want to decouple parts of the execution of process and have full control over the environment this might by ok.

But when other players enter the game this situation might not be so convenient. The data format is just a data structure (which doesn’t have any meaning anywhere other than RPG where we could share the data structure via a copy book). The encoding (EBCDIC) is also not very mainstream and thus not necessarily generally available on other platforms / programming languages.

If a Java program needs to receive an auction message it could use the JTOpen project for getting things together (data format and encoding) but that would be rather tedious.

If a Node.js program needs to receive this message you would be rather hard pressed to get things to work.

Solution

One possible solution would be to send the data as a JSON string. JSON is a very universal format and has the default encoding of UTF-8 which is also widely available. Thus you are not limited to your own platform but are also able to invite other players into the game without having to change anything on your side.

Multiple Formats

On the receiving side we might want to support different formats or versions of the data. One way to implement this would be to publish the format on you intranet / extranet in some way and use a custom content type. Content types are not limit by the strings we see so often on the internet like application/json or text/html. You can use whatever you want though there are some best practices, see Media Type Specifications.

If we want to convey that this is an auction we could use the content type application/x.rpgnextgen.auction. There is an established way of adding additional information like format and version to the content type by appending the info with an add sign.

1	application/x.rpgnextgen.auction+json+v1

Now the receiver knows pretty well from the message alone what he can expect of the message body.

We can query the frame header by using stomp_frame_getHeader.

1	stomp_frame_getHeader(frame : 'content-type');

JSON

The noxDB project in its master branch supports parsing a JSON string from a pointer variable out-of-the-box. We can just pass the pointer from the frame buffer to noxDB.

dcl-ds frameBuffer likeds(stomp_util_buffer_t);
dcl-s json pointer;

...

frame = stomp_receiveFrame(client);

json = jx_parseString(frameBuffer.data);

jx_close(json);

In the utf8 branch which is packaged for use with iPKG we need to take a little extra effort into account as we cannot directly pass the pointer. We need to pass the data into a string variable and pass that extra variable to the noxDB procedure.

dcl-ds frameBuffer likeds(stomp_util_buffer_t);
dcl-s json pointer;
dcl-s string like(UTF8);

...

frame = stomp_receiveFrame(client);
frameBuffer = stomp_frame_getBody(frame);
string = %str(frameBuffer.data : frameBuffer.length);
json = nox_parseString(string);

...

jx_close(json);

API Documentation

You can find the API documentation of the STOMP for ILE project and noxDB project at iledocs.rpgnextgen.com. For those interested in generating API documentation for ILE programs and service programs: The documentation at iledocs.rpgnextgen.com has been generated with the project iledocs-page-builder.

Happy integrating!

Mihael

i means integration - Sending data with STOMP client

Sun, 12 Nov 2023 00:00:00 +0100

In the last posts we have set up all necessary building blocks for sending data to a message queue on the Apache Artemis instance on IBM i. Now we are putting all those pieces together and write an ILE RPG progam which sends data to a message queue with the STOMP service program.

We are picking up the case from the first part of this blog series. We want to inform that a new auction has started including the items that will be part of the auction.

1  **FREE
2
3  ctl-opt main(main);
4
5  /include 'stomp.rpginc'
6
7  dcl-ds auction_t qualified template;
8     id uns(20);
9     title varchar(100);
10     description varchar(1000);
11     items likeds(auction_item_t) dim(10);
12     auctionStart timestamp;
13     auctionEnd timestamp;
14 end-ds;
15
16 dcl-ds auction_item_t qualified template;
17     id uns(10);
18     name varchar(100);
19     quantity uns(3);
20 end-ds;
21
22 dcl-proc main;
23     dcl-s client pointer;
24     dcl-ds auction likeds(auction_t);
25
26     auction = fillAuctionData();
27    
28     client = stomp_create('localhost' : 40002);
29     stomp_open(client);
30     stomp_command_connect(client : 'auction_user' : 'uwillneverknow');
31     stomp_command_send(client : 'auction-events' : %addr(auction) : %size(auction) : STOMP_MIME_BINARY);
32     stomp_command_disconnect(client);
33     stomp_close(client);   
34     
35     on-exit;
36         stomp_finalize(client);
37 end-proc;

This looks like a lot of code but the main procedure is just 16 lines. So it is not much at all. Let’s look at the code line by line.

Line 1 : We are writing fully free RPG code. We are not half-assing this already from the start ;-)
Line 3 : We will compile this program in two steps. Thus we only declare the main procedure in the control options.
Line 5 : The STOMP copybook needs to be included so that we can use the constants and prototypes from the STOMP project. Everything else will be added to the compile commands.
Line 7 - 20 : The definition of the auction data which will be sent to the message queue.
Line 26 : The data is loaded here. Nothing fancy.
Line 28 : An instance of the STOMP client is created. The client is accessed via the pointer. The internal data of client doesn’t matter to us. We are only using the procedures from the STOMP project anyway. The host and port we want to connect to are passed to the create procedure. The port should be the same we used in the Artemis configuration for our STOMP acceptor.
Line 29 : A socket connection to the message broker is created.
Line 30 : To connect on a (STOMP) protocol level we need to send a STOMP frame/command to the server. This also includes the credentials used to login to the message broker. For an anonymous connect we can just pass the client.
Line 31 : Now here we are sending the data at last. It will be sent as a block of bytes. The MIME type also states that it is no text data but raw bytes which are the payload of the message.
Line 32 : If no more message are to be sent the connection can be closed (on a protocol level).
Line 33 : The socket connection needs to be closed.
Line 36 : Last but not least the memory allocated by the client needs to be freed.

We can see the message in the sink queue we configured in the Artemis server. But there are some suboptimal things we can improve.

Character Encoding

The data sent to the message queue is in the CCSID of the current job. By default this is stated nowhere in the message. So the receiver of the message has no idea what the encoding of the message content is. One way of dealing with this is adding the charset parameter to the Content-Type with the correct charset for the current CCSID, like application/octet-stream; charset=x-ebcdic-germany.

But EBCDIC is not a common character encoding and thus not a good encoding for data exchange as the sender doesn’t know who will receive the message.

Sending the message in the JSON format

The message we sent in the above code is hard to read as is because it was sent as a sequence of bytes. We used the data structure directly as the format of the data sent to the message queue so every message receiver must also know the definition of the used data structure. As this is RPG specific it is not a good choice for data exchange. It would be better to use a data format which is not programming language specific … like JSON.

Currently many services in the whole world exchange data in JSON format. We will also use this as it is a very common and nice format which is also easily readably by humans.

This also solves the “problem” with the character encoding as JSON is by default encoded in UTF-8.

noxDB

On IBM i there are several ways to generate JSON. Personally I like the noxDB project. It is a fast and flexible JSON software library available as a ILE service program. It supports the data-gen opcode to easily generate JSON based on a data structure but also lets you create a fined tuned JSON document if you want to. So you have total control over the JSON generation. There are also some gems in the library like generating JSON based on a SQL result set.

The project comes in two flavors: version 1 = EBCDIC based , version 2 = UTF-8 based

The version 2 (UTF-8 based) is packaged as an RPM package and available in the RPM repository at RPG Next Gen. It can be installed with the iPKG client. The installation is as easy as always:

1 2	ipkg install noxdb2 ipkg install 'noxdb2-devel' loc('/home/mihael/include')

I won’t go into the details of how the noxDB service program can be used. There are dozens of examples.

stomp_command_send

We will rather take a look at the procedure that sends the data: stomp_command_send

dcl-pr stomp_command_send extproc(*dclcase);
    client pointer const;
    destination varchar(100) const;
    messageData pointer const options(*string);
    messageLength uns(10) const options(*omit : *nopass);
    contentType varchar(100) const options(*omit : *nopass);
    headers pointer const options(*nopass);
end-pr;

Client

As with most of the procedures we need to pass the client “object” (the pointer) to the procedure.

Destination

Second we need to tell the procedure to which queue the message should be sent. This name maps directly to a address name in Apache Artemis. In our case this is auction-events.

Message Data

We notice that the parameter for the message data is a pointer with options(*string). The procedure can work with a null-terminated string (meaning a string with the termination character x'00' at the end of it). It expects a null-terminated string if no message length is provided.

In RPG this means that we can declare a variable char or varchar and end the data with x'00'.

dcl-s data varchar(100) ccsid(*utf8);

data = '{ "id" : 358 }' + x'00';

stomp_command_send(client : 'auction-events' : %addr(data : *data));

But as options(*string) has been declared on the paramter we can also just pass the variable. No need to bother with any C-style string stuff.

dcl-s data varchar(100) ccsid(*utf8);

data = '{ "id" : 358 }';

stomp_command_send(client : 'auction-events' : data);

This looks much better. :)

It is also possible to pass a string literal.

1	stomp_command_send(client : 'auction-events' : '{ "id" : 358 }');

Message Length

Without passing the message length the procedure will try to determine the length by searching for the first x’00’ value. This is not always very convenient. If you want to send binary data and the data might have x'00' as a value then you need to pass the length of the data as a parameter.

Content Type

Similar to HTTP message we also should specify the Content-Type of the data we want to send so that the receiver knows how to interpret the data.

If we always want to the send the same type of data (meaning also the same Content-Type) then we can set the content type on the client and don’t have to pass it on every call.

1	stomp_setContentType(client : 'application/prs.rpgnextgen.auction')

The content type passed to the send procedure will overwrite any global settings on the client.

Headers

If we want some additional headers on the frame we send to the server we can pass them to the send procedure.

The headers parameter is declared as a pointer. This doesn’t tell us much. But it expects a list created with the Linked List service program. Each entry needs to be in the format of the data structure stomp_frame_header_t.

dcl-s headers pointer;
dcl-ds header likeds(stomp_frame_header_t) inz;

headers = list_create();
header.key = 'server';
header.value = 'ibmi.localhost';
list_add(headers : %addr(header) : %size(header));

...

stomp_command_send(client : 'auction-events' : %addr(auction) : %size(auction) :
    STOMP_MIME_BINARY : headers);

...

on-exit;
    list_dispose(headers);

Don’t forget to free the allocated memory of the list by calling list_dispose(headers).

Receipts

To make sure that the server has successfully received the message sent to the server we can request to acknowledge the receiving of a message with a receipt. So for every message we send to the server the server sends back a RECEIPT frame to tell the client that the message has been successfully received.

1	stomp_useReceipts(client : *on);

Note: Receipts only make sure that the server successfully received the message. It does not mean that the message has been successfully distributed to any other client.

Mark message as persistent

By default messages sent via the STOMP protocol are not persistent, meaning they will vanish if the server is restarted. STOMP also supports persistent / durable messages. This can be achieved by adding the header persistent to a message, see STOMP_OPTION_PERSISTENT which can be set on the STOMP client instance.

1	stomp_setPersistMessages(client : *on);

API Documentation

You can find the API documentation of the STOMP for ILE project at iledocs.rpgnextgen.com. For those interested in generating API documentation for ILE programs and service programs: The documentation at iledocs.rpgnextgen.com has been generated with the project iledocs-page-builder.

Happy integrating!

Mihael

i means integration - STOMP

Mon, 23 Oct 2023 00:00:00 +0200

I covered how to install Apache Artemis on IBM i in the last part of this series. Now we need a way to connect to Artemis from our ILE environment as this is the main environment in classic SMB IBM i shops.

Lets take one step back and have a look at the internet where many different platforms and programming languages work together. HTTP is one of the main pillars of the world wide web. It is a rather simple protocol (at least version 1.0 and 1.1) which can be implemented in probably any programming language.

There is something similar in the messaging world: STOMP - Simple Text Orientated Messaging Protocol. It has many similarities with HTTP and is also easy to implement. And most messaging middlewares support this protocol and so does Artemis. We already configured one acceptor in Artemis especially for the RPG client in the previous post. The acronym STOMP hints that the protocol can only be used with text message but that is not the case. You can also use STOMP for sending binary data.

STOMP Installation - online

Luckily there is already an RPG implementation of the STOMP 1.2 protocol ready to use at RPG Next Gen. It has been conveniently packaged for an easy installation with iPKG. We just need to install the iPKG client (just download the save file and restore the two objects) and are good to go to install the STOMP service program and the corresponding RPG copybooks for all the prototypes.

iPKG is an ILE program and command and can be executed from the 5250 terminal session.

First we need to create a library where we want to install the objects. Then we need tell iPKG where to look for packages. We do this by registering the RPM repository from RPG Next Gen.

1 2	crtlib ipkg ipkg addrepo 'RPGNextGen https://repo.rpgnextgen.com/repository'

iPKG needs to download and process the meta data from this repository. This is done by executing the command update.

1	ipkg update

The STOMP service program has the following software packages as dependencies:

Linked List
Message
libtree

All these packages are included in the RPG Next Gen RPM repository and can be installed with iPKG.

1
2
3

ipkg install linkedlist
ipkg install message
ipkg install libtree

After successfully installing the dependencies we can install the STOMP service program.

1	ipkg install stomp

For using the STOMP service program we need the prototypes and constants of the STOMP project. These are provided as copybooks in stream files and are packaged for iPKG available in the RPG Next Gen RPM repository.

Packages which contain stuff necessary for development like prototypes and header files have the suffix “-devel“ like stomp-devel. By default the stream files will be extracted to the folder /usr/local/include. But we can also specify a different location with the parameter LOC

1	ipkg install 'stomp-devel' loc('/home/mihael/include')

Now we have everything we need for building our first program with the STOMP service program.

STOMP Installation - offline

For everybody with an IBM i server with no or restricted access to the internet: iPKG can also be used offline. You will still need to get the RPM packages from the internet (either the single packages or the whole repository). But you can do this yourself and put the repository or packages on the IFS on your IBM i server or host these packages somewhere inside your network on a web server.

RPG Next Gen Repository

The RPM repository at RPG Next Gen offers the repository packaged for offline usage. Once you have extracted the repository from the archive file you can use it with the iPKG command and install packages as usual.

You first need to register the repository and update the meta data.

1 2	ipkg addrepo 'RPGNextGen file:///home/user/repository' ipkg update

Now you are free to install any packages you need from this repository just by using the package name on the install command.

1	ipkg install stomp

Single packages

The other option is to download the packages you are interested in one by one to the IFS and install them. The install command not only accepts a package name but also a RPM file name and a path to a RPM file. The RPG Next Gen RPM repository lists the available packages in the RPM Packages section.

If you placed the RPM files at /home/user/rpm and want to install the package STOMP version 2.0.0-2 you can execute the following command.

1	ipkg install '/home/user/rpm/stomp-2.0.0-2-ppc-ibmi-7.2.rpm'

If you have a couple of packages you want to install by this way you can first change the current directory with CHGCURDIR '/home/user/rpm' and then just state the file name on the install command like this:

1	ipkg install 'stomp-2.0.0-2-ppc-ibmi-7.2.rpm'

STOMP Installation - compile

Of course you can also compile the STOMP project yourself. It is not too hard but requires some effort as you need to also manually compile the dependencies in the correct version.

Feel free ;-)

iPKG Notes

All the shown iPKG commands expect that we use the library IPKG as our target library. If we want to use another library for this we can add the parameter IPKGLIB(MY_TST_LIB) to all the ipkg commands. You can use as many libraries with different repositories and/or installed packages as you like. Each target library is self-contained and does not mess with any other installation.

All service programs provided by the RPG Next Gen RPM repository will automatically be added to the IPKG binding directory. The binding directory is located in the library specified in the IPKGLIB parameter.

When installing packages with objects from the QSYS.LIB file system iPKG will execute a RSTOBJ command. The user installing the packages needs to have enough authorities to use this command.

For any further infos on iPKG take a look at the project wiki or contact me.

Happy integrating!

Mihael

i means integration - Apache Artemis

Sat, 14 Oct 2023 00:00:00 +0200

On the previous post I mentioned some pros for decoupling programs. One way of doing this is by using messages and message queues. One program puts a message on a queue and another program receives that message and thus informing the program that it has to do something. There are many middleware software packages which support this.

One of those packages is Apache Artemis. It supports one-to-one connections (queues) but also one-to-many (topics). The good thing about Apache Artemis is that it is written in Java and thus can run on IBM i. Just download the archive to your IBM i and extract the content of the archive (f. e. with unzip apache-artemis-2.31.0-bin.zip or jar -xf apache-artemis-2.31.0-bin.zip).

I like to place the software I manually install on the IFS on IBM i in the /opt folder. In addition I like to create a symlink (ln -s ...) so that it is easier to navigate the folders (but that is just a personal habit ;-) ). So my folder structure looks like this:

1 2	/opt/apache-artemis-2.27.1 /opt/artemis -> apache-artemis-2.27.1

Apache Artemis has a very good documentation and they explain everything in detail. Thus here an excerpt from it explaining how to install a new Apache Artemis broker instance.

The ActiveMQ Artemis broker follows a different paradigm where the project distribution serves as the broker “home” and one or more broker “instances” are created which reference the “home” for resources (e.g. jar files) which can be safely shared between broker instances. Therefore, an instance of the broker must be created before it can be run. This may seem like an overhead at first glance, but it becomes very practical when updating to a new Artemis version for example.

To create an Artemis broker instance navigate into the Artemis home folder and run: ./bin/artemis create /path/to/myBrokerInstance on the command line.

Let us place the Artemis broker instances at /var/local/lib/artemis/<broker>. The broker will manage auction messages so my command line looks like

1	./bin/artemis create /var/local/lib/artemis/auction

Now I could run this Artemis instance by executing /var/local/lib/artemis/auction/bin/artemis run. But first I need to configure some things. All configuration files of an Artemis instance can be found at the instance location in the folder etc.

mihael@ibmi:~/auction-broker/etc> ls -1
artemis-roles.properties
artemis-users.properties
artemis.profile
bootstrap.xml
broker.xml
jolokia-access.xml
log4j2.properties
login.config
management.xml

Acceptors

Acceptors are the things that make the network connections to the clients and receive/distribute messages. Artemis supports many protocols. But the protocol we are interested in is STOMP. Artemis can be configured to have many acceptors listening on different ports but also having one acceptor accepting multiple protocols listening on one port.

Note: As I need some “special” connection attributes for the local STOMP acceptor I go for one acceptor for local STOMP connections and one acceptor for everything else.

Here is the acceptor configuration in broker.xml.

<acceptors>
    <!-- Acceptor for every supported protocol available locally and on the network -->
    <acceptor name="artemis">tcp://0.0.0.0:40001?tcpSendBufferSize=1048576;tcpReceiveBufferSize=1048576;amqpMinLargeMessageSize=102400;protocols=CORE,AMQP,STOMP,HORNETQ,MQTT,OPENWIRE;useEpoll=true;amqpCredits=1000;amqpLowCredits=300;amqpDuplicateDetection=true;supportAdvisory=false;suppressInternalManagementObjects=false</acceptor>

    <!-- 
        STOMP acceptor only available on localhost with extra long connection timeout. 
        connection TTL: 60 * 60 * 24 * 1000 = 86400000 milliseconds = one day
    -->
    <acceptor name="stomp">tcp://localhost:40002?tcpSendBufferSize=1048576;tcpReceiveBufferSize=1048576;protocols=STOMP;useEpoll=true;connectionTtl=86400000</acceptor>
</acceptors>

Note: The connection TTL attribute is set because the RPG STOMP service program does not support sending heart beats at the moment and Apache Artemis does not accept having no heart beat (and I wouldn’t either ;-) ) so we are setting a very high connection TTL so that it is guaranteed that some event will be sent in this time frame (f. e. a day or a week).

Topic

Queues and topics can be automatically created but I also can specify a persistent topic in the configuration.

broker.xml

<addresses>
  <address name="auction-events">
    <multicast>
      <queue name="sink">
        <durable>true</durable>
      </queue>
    </multicast>
  </address>
</addresses>

Here we have a topic auction-events and even if there is no subscriber to that topic all messages will at least be distributed to the queue sink which is really convenient in the development phase. So even if nobody subscribed to auction-events we will still see the incoming messages in the queue sink.

Logging

From the documentation:

Apache ActiveMQ Artemis uses the SLF4J logging facade for logging, with the broker assembly providing Log4J2 as the logging implementation. This is configurable via the log4j2.properties file found in the broker instance etc directory, which is configured by default to log to both the console and to a file.

I couldn’t have written it better.

For logging everything related to the STOMP acceptor just raise the log level of the StompConnection logger.

1	logger.org.apache.activemq.artemis.core.protocol.stomp.StompConnection.level=DEBUG

Note: You may need to also raise the level of the corresponding appender to DEBUG.

1 2	appender.console.filter.threshold.type = ThresholdFilter appender.console.filter.threshold.level = DEBUG

Web Console

Apache Artemis comes out-of-the-box with a web console based on Hawtio. The only problem with this is that it is only available on the local machine. But just for giving this a try it would be easier if we could access it from outside localhost. For this we need to change two configuration entries.

bootstrap.xml

<web ... >
  <binding name="artemis" uri="http://0.0.0.0:40000">
    ...
  </binding>  
</web>

jolokia-access.xml

<cors>
  <allow-origin>*</allow-origin>
  ...
</cors>

Now we can access the Apache Artemis web console running on IBM i from our computer by accessing the URL http://my_ibmi:40000.

Large Text Messages

By default the web console will show you only the first 256 characters from a message. Depending on the project messages might be much larger (several thousand characters). The max. number of characters shown in the web console can be set in the broker.xml file.

broker.xml

<address-settings>
  <address-setting>
    ...
    <management-message-attribute-size-limit>50000</management-message-attribute-size-limit>
    ...
  </address-setting>
</address-settings>

Here the max. number of displayed characters is set to 50.000.

Starting Artemis

Now I can run this Artemis instance by executing /var/local/lib/artemis/auction/bin/artemis run and have the log output in /var/local/lib/artemis/auction/log/artemis.log.

But there are other ways to start Artemis, f. e. via Service Commander.

SocketException on Startup

Depending on the network configuration of the IBM i server we may encounter the following error on startup:

1	java.net.SocketException: Protocol driver not attached

This error may occur on any product/project which utilizes Jetty. But we can work around this problem by specifying a Java system property. The startup script of Artemis supports adding Java system properties without having to modify the script by using the environment variable JAVA_ARGS_APPEND.

1	export JAVA_ARGS_APPEND="-Djava.net.preferIPv4Stack=true"

Now we can start Artemis as usual.

Service Commander

Service Commander is a tool for managing services and applications on IBM i. But in contrast to other tools Service Commander can manage services and applications of both worlds, PASE and native IBM i, and has a very nice integration with the native CLI by utilizing STRTCPSVR.

Service Commander stores the necessary data for each managed service in a stream file on the IFS, see folder .sc/services in the home folder. I placed the following configuration at /home/myuser/.sc/services/artemis.yaml:

name: Artemis
dir: /var/local/lib/artemis/auction
start_cmd: /var/local/lib/artemis/auction/bin/artemis run
stop_cmd: /var/local/lib/artemis/auction/bin/artemis stop

check_alive: 40000,40001,40002

batch_mode: yes
sbmjob_jobname: ARTEMIS
sbmjob_opts: JOBQ(QUSRNOMAX) USER(PM)

environment_vars:
  - "JAVA_HOME=/QOpenSys/pkgs/lib/jvm/openjdk-11"
  - "QIBM_MULTI_THREADED=Y"
  - "JAVA_ARGS_APPEND=-Djava.net.preferIPv4Stack=true"

Now we have a whole bunch of possibilities for starting our Apache Artemis instance.

From a SSH session: nohup /var/local/lib/artemis/auction/bin/artemis run > /dev/null 2> /dev/null &
From a SSH session: sc start artemis
From 5250 session: STRTCPSVR SERVICE(*SC) INSTANCE('artemis')

And now we can start it equally well from the PASE and the 5250 environment. And also no problems with having it autostarted after an IPL. Just add it to the autostart group in the Service Commander job definition YAML file.

Kudos to Jesse Gorzinski for this great project!

Happy integrating!

Mihael

PS: I mentioned “… written in Java … thus can run on IBM i.”. That is not totally correct. Because it is written in Java it can run in the Java VM on IBM i. But that does not guarantee that it will run correctly on IBM i. Many Java projects are done in mind with x86 as their target platform (or even only Windows) and thus it does not always work on IBM i. But so far I had luck with all of the Java projects I tried on IBM i. :-)

i means integration - part 1

Wed, 11 Oct 2023 00:00:00 +0200

As many know I am a proponent of the platform IBM i. Many see it as an old legacy platform which cannot play in the modern world of computing. Nothing could be farther from the truth. It is one of the most advanced platforms out there.

And what about the programming languages which has been with the platform since its beginnings? RPG has evolved into a modern and one of best programming languages for business logic. Because of its ILE capabilities it has access to the whole ILE environment including ILE C and C++ modules and libraries. And thus it can play in any game it needs to play … also in the web and micro service game of today.

But first let us have a look at the following case.

The Case

We have an online auction house for our famous online game. The game started decades ago where we only had to cover ingame trades. We have a program for managing these trades: ACT001. It covers all the necessary attributes of an ingame auction.

And then … things started to evolve. Other media channels (like WhatsApp, Discord or Telegram) sprang up and we want to take advantage of these new possibilities. We like to inform players about new auctions of interest and so we need to provide the interface to these media channels with data. There is also a player by the name of “extranet” which needs some of the data.

So we have more information to manage which might evolve into multiple programs for managing data and we have different channels where data enters our system. Additionally we have multiple systems and interfaces which need to be provisioned with our data.

After some years most RPG programs look like this:

begsr savact;
  ...

  write actdtaf01 data;

  call act002; // WhatsApp
  call act003; // Telegram
  call act004; // Extranet

  ...
endsr;

And the situation is more like this:

Every auction data management program needs to call every distribution program which leads to a maintenance nightmare. One solution would be to decouple things.

Decoupling : Pros and Cons

Pros for decoupling components (programs):

defined interface between components
easier to extend
easier to maintain
easier to scale
easier to handle errors (and replay events)
… and some more

Of course like with everything else in life there are not only pros but also cons like

you cannot derive from the management program where the data will flow to
some events must be processed in the correct order
may need more infrastructure (software)
… and probably some more

But in my opinion the pros outweight the cons big time.

Solution

Some may say:

That is pretty easy. Put the calling of the distribution program in a trigger. It will be executed every time anything changes.

Yes. That may be a solution. But in case a distribution program is called and the call results in an error what will happen to the update of the data? Will it be rolled back? But there is nothing wrong with the data. Perhaps the receiving system is not online.

Why should we not update our own system and move on?

We could do that but then what about the update of the other system that needs our data? Now we need a way to handle the error case(s) … for every distribution program.

So a trigger may be one viable solution … but there are also others solutions.

Another solution

If we distribute our data to other systems it may not be needed there immediately and may not always need to be in sync with our own system. So most of the time we may be able to do the distribution asynchronously.

Synchronously : We tell someone that our data has changed and watch that someone do his work with our data to update his system.

Asynchronously with one receiver: We tell someone that our data has changed and he should update his system. We don’t wait and watch his update but just move on and do our work.

Asynchronously with multiple receiver: We tell someone that our data has changed and that he should inform everybody who is interested in the data change. And we just move on.

The one-to-one situation of data publisher and data receiver can already be done with little effort natively on IBM i.

The magic keywords are: data queues!

They are easy to manage and use and are very performant. But in our case they are of little help as we have many different data receivers.

For the 1-to-n situation we can use a concept called pub/sub. One publisher publishes an event and multiple subscribers may subscribe on this type of event. This concept is quite generic and may be used pretty much everywhere. Frontend, backend … it doesn’t matter.

Does this solution rules out the use of a database trigger: no. Acutally a trigger might be a good way to trigger this event and act as the data/event publisher.

There is currently no ILE solution for pub/sub that I know of. But that does not matter so much as we want to do things in an asynchronous way anyway (pretty much fire and forget) and thus it does not matter if it is ILE or not. The only thing which should be ILE compatible is the data publisher. And we got that covered with an open source service program available on rpgnextgen.com.

There are many systems available which cover the pub/sub concept. A very famous, stable and widely used system is part of the Apache software ecosystem: Apache Artemis.

A good thing about Apache Artemis is that it is written in Java and thus can run on IBM i without any problems.

In the next episode we take a look at Apache Artemis and how to run it on our beloved IBM i.

Happy integrating!

Mihael

Jenkins on IBM i - Part 1

Sun, 12 Feb 2023 00:00:00 +0100

Some time ago (probably some years) I wrote a wiki article “IBM i and Continuous Integration”. I was kind of successful with my adventure in the realm of CI and DevOps to some degree. I could install the CI server but had some trouble with running the commands on IBM i.

Some years and software versions later I am giving it a try again. I think CI is an important part in software development and not so hard to implement (at least when it comes to the basics).

So what is the goal?

Continuous Integration!

But what is “Continuous Integration”? Again I will refer to the Wikipedia article Continous integration. I think the main point is this:

… the practice of merging all developers’ working copies to a shared mainline several times a day that triggers automated build with testing …

And today this merging can be done even easier with the version control systems of today which can trigger a build automatically after a successful merge.

Why CI even if you are a single developer in your shop?

Having a build server can be handy for many reasons:

Build server can be setup to have the same environment as the production environment. The development environment often differs from the production environment.
Every build is done in exactly the same way.
Builds can be triggered regardless of the developer machine.
Release builds
… and there are many many more

Here some quotes from the net:

The CI build can also be thought of as your “release” build. The environment should be stable, and unaffected by whatever development gizmo you just add to your machine. It should allow you to always reproduce a build. This can be valuable if you add a new dependency to your project, and forget to setup the release build environment to take that into account. — Is Continuous Integration important for a solo developer?

Likewise, if you are building libs that are used by multiple projects then CI will make sure they work with ALL of the projects rather than just the one that you’re working with right now… — Is Continuous Integration important for a solo developer?

So … Continuous integration is for everyone!

Task

We will take a RPG project with the source located in a Git repository and will build it with Jenkins on an IBM i server. The build will be executed with the tool make. We will see what Jenkins and its’ plugins can bring to the table.

Build Server

Nothing has changed here. I am still a big fan of Jenkins. Jenkins is written in Java and runs very well on IBM i without any modifications. You should install Java 11 on your IBM i either via yum or as a licensed product.

We can test the java installation with executing java -version on the shell. Your output in a shell should look something like this:

-bash-5.1$ java -version
openjdk version "11.0.11-ea" 2021-04-20
OpenJDK Runtime Environment (build 11.0.11-ea+9)
Eclipse OpenJ9 VM (build openj9-0.26.0, JRE 11 OS400 ppc64-64-Bit Compressed References 20220208_964 (JIT enabled, AOT enabled)
OpenJ9   - b4cc246
OMR      - 162e6f7
JCL      - 7796c80 based on jdk-11.0.11+9)

The important part is the version “11.0.11-ea”. It should at least state version 11. Java in general is very backwards compatible. A more recent version than 11 should work without any problems.

Installation

Jenkins can be downloaded from the Jenkins website.

The installation is pretty straight forward. Jenkins is packaged as a war file but also supports executing the file as a normal Java application.

The easiest way to start Jenkins is from a shell (f. e. bash via a SSH connection):

1	java -jar jenkins.war

The Jenkins project has an extensive documentation. The website provides a page Installing Jenkins. The WAR file section is probably the documented installation platform which maps best to IBM i PASE (AIX).

The Jenkins runtime can be configured via command line parameters and Java system properties. A list of supported command line parameters (f. e. HTTP port) can also be found here. The extensive list of Java system properties can be found here.

By default all the data will be stored in the .jenkins folder in the home directory of the current user and the default port is 8080.

This might not be convenient for your environment and we can change that to some other values.

1 2	export JENKINS_HOME=/var/local/lib/jenkins java -jar jenkins.war --httpPort 5000

Now Jenkins uses port 5000 and stores its data in the folder /var/local/lib/jenkins.

Note: You may need to add the Java system property “java.net.preferIPv4Stack=true” when you see Jetty choking out some weird network exception. So your command line then looks like java -Djava.net.preferIPv4Stack=true -jar jenkins.war --httpPort=5000.

Configuration

The path to the tools used in the build jobs need to either be specified in Jenkins or need to be in the PATH environment variable.

The PATH environment variable can be configured in Jenkins management section.

1	Dashboard ⇒ Manage Jenkins ⇒ Configure System ⇒ Global Properties

The location of some essential tools can be configured extra in

1	Dashboard ⇒ Manage Jenkins ⇒ Global Tool Configuration

If you don’t have git in the PATH you can set it here to /QOpenSys/pkgs/bin/git.

Build Job

A Build Job defines what is executed and also where and when. It defines the execution environment. In Jenkins you can configure a job in different ways. I will cover “Freestyle Project” and “Pipeline”. Freestyle projects consist of predefined build steps which can be concated to a workflow. Pipeline projects are mainly defined by a pipeline script which is written in a specific DSL. Pipeline scripts can be written in two different styles: scripted and declarative. In the declarative style you use mainly predefined functions and don’t script as much yourself (though you also have the option to script in the declarative pipeline). The Pipeline project will be covered in more detail in another post. Here I will focus on the “Freestyle” project.

Freestyle Project

Build Jobs can be configured in Jenkins in Dashboard ⇒ New Item. Configure the job as a “Freestyle Project”. Freestyle projects are configured in the UI with build steps and configuration options contributed by other Jenkin plugins.

The workflow of a “freestyle” build consists of steps. Steps can either be added as “Build Steps” or “Post-Build Steps” by using the corresponding dropdown button in the configuration of the build job.

Git

In a freestyle project there is a predefined option to check out the source code from a source code repository. Here we can enter our URL for the Git repository.

Compile

This project uses make for building the project. We first execute the purge target to have a clean build environment and then the all target. We specify the target library as a parameter of the gmake command, see BIN_LIB.

1	gmake BIN_LIB=MIHAELSTMP purge all

Note: I am using gmake which can be installed via yum which is the GNU version of make.

We add a build step “Execute shell”.

What happens on a failed compile?

Easy. gmake will return a non zero exit value if it cannot execute a target successfully. By default the shell step will mark the build as failed when it gets a non zero exit value from the executed shell command. So on a failed compile the build will automatically be marked as failed.

Open Tasks

In the old wiki article I mentioned the Task Scanner Plugin for scanning the source code for specific keywords which mark open tasks. That was a couple of years ago. In the meantime many plugins have been merged into the Warnings Next Generation plugin which also now includes the feature set of the Task Scanner plugin.

In a freestyle project the open task scanning can be added as a post-build action by selecting “Record compiler warnings and static analysis results”.

The result can be found by selecting the menu entry “Open Tasks” on the main page of the executed build.

Notification

Having an automated build is pretty cool. But what about the results. In most cases we would want to know the result of the build. But we don’t want to stare at the screen and wait for the result. We want to be informed by Jenkins when the build has finished.

Jenkins has many plugins which cover the different ways of communicating the result of a build. But the most simple way is to get an email. We can use the default email plugin or use the extended email plugin. Both plugins can be configured under Dashboard ⇒ Manage Jenkins ⇒ Configure System.

But just installing and configuring the email plugin is not enough. That will not trigger an email on a finished build. We need to add a Post-Build action step like “E-mail Notification” or “Editable Email Notification”.

Starting Jenkins

At the end I would like to introduce you to another option of starting Jenkins on IBM i. The project Service Commander from Jesse Gorzinski does a wonderful job of managing services and keeps configuring the environment of those services simple. Especially the Java and Node.js applications make heavy use of environment variables and that is one area where Service Commander really shines.

I don’t want to go into the details of how to install and use Service Commander because Jesse already did a great job of that on the project Github site. Just read the README file. But I want to share my start script with you.

name: Jenkins
dir: /home/mihael/jenkins
start_cmd: java -Djava.net.preferIPv4Stack=true -jar jenkins.war --httpPort=5000

check_alive: 5000
batch_mode: true
sbmjob_jobname: JENKINS
sbmjob_opts: JOBQ(QUSRNOMAX)

environment_vars:
- HOME=/home/mihael
- JAVA_HOME=/QOpenSys/pkgs/lib/jvm/openjdk-11
- PATH=/QOpenSys/pkgs/bin:/QOpenSys/usr/bin:/usr/ccs/bin:/QOpenSys/usr/bin/X11:/usr/sbin:.:/usr/bin

The beauty of this is that you can define your environment without any hassle and still integrate with the traditional management of jobs on the IBM i server. This Jenkins instance will have JENKINS as its job name and thus can be easily found.

Kudos to Jesse!

Wrap up

With a couple of steps we can have quite a smooth development lifecycle which automates many of those tedious steps and let us play it safe with source code scanning and unit testing … in a team or as a lone wolf.

There are still some areas which we haven’t covered in detail: unit testing and documentation. Perhaps this will be covered in a future post.

Jenkins also supports triggering a build by a HTTP request. But as this post is already very lengthy this topic may be covered in another post.

Happy building!

Mihael

Cool things you might not know about RPG or ILE

Fri, 18 Nov 2022 00:00:00 +0100

Long time no see! :) . It has been a while since my last post. Here is a little different post. I compiled a list of things I really like a lot about RPG and the ILE environment and am missing (most of the time) in other programming languages / environments.

Note: The features in the list are in no special order.

Database connections — This one is a real no brainer. I have encountered no other programming language or environment where it is so easy to get access to the database as on IBM i and with RPG. No connection pooling, no connection management. Just “always on” :) !
Database operations — This is also about the database but more about how to work with / access the database from RPG and not about the connection. In RPG we can choose between the built-in support for Record-Level-Access and embedded SQL. No extra library to load. Just working out of the box.
Data types — RPG has a rich collection of data types. Integers and chars are covered by every programming language but you probably won’t find a numeric data type with fixed decimal positions. Date, time and timestamp are also very nice to have and especially the operations you can do with them in RPG.
Pointers — Not widely used in RPG but pointer support is really nice. Especially the feature that you can declare a variable based on a pointer.
User heaps — User defined memory heaps are really cool when you have to deal a lot with dynamic memory management. You don’t necessarily have to keep track of all the allocated memory. Just discard the heap and everything is cleaned up for you by the system. No keeping track of pointers. Another cool feature is that you can mark a memory allocation in the heap and later free up everything up to that mark.
User spaces — User spaces also exist on other platforms but I think they don’t have so many features there. On IBM i user spaces are real objects on the file system with all features of an object in the QSYS.LIB file system. They are also persistent between reboots.
Data areas — These easy to use objects are great for storing simple data. They have native support for editing and displaying the data and are fully integrated into the RPG programming language. On other platforms you would probably use a normal text file. But reading, locking and writing a text file on other platforms is not as easy as in RPG.
CCSID conversion — Another great underrated feature is the CCSID support in RPG. You can declare a character variable with any CCSID and assign a value to it. This value will be automatically converted to the CCSID of the declared variable. No extra step needed. So for converting a value with EBCDIC CCSID 37 to UTF-8 is only declaration of two variables and the assignment of the value. That’s it.
Compile time data — This is usually used to initialize arrays. It can easily be achieved by other means in other programming languages but I found a liking to how it is done in RPG.
Polyglot — One of the biggest things in the ILE environment IMO is the possiblitly to bind multiple modules coded in different programming languages together to a single program object. Coupled with the fact that one can use any exported symbol (function, procedure, data) from any ILE capable language makes way for many possiblities and let you choose the right language for the task. F. e. C for networking stuff and RPG for business logic and data processing. ILEastic is my favorite example for demonstrating this.
Thread safety — Most of the time the normal RPG developer won’t have to bother about threads and thread safety. But if your application or framework makes use of multiple threads you need to make sure that your code is thread safe. This is super easy in RPG. Just add a control option to your code like this:
1
ctl-opt thread(*concurrent);
Note: If you are using embedded SQL and transactions this control option will not be enough because transactions are scoped to the job or activation group. One solution to this is to activate the SQL Server Mode with the QWTCHGJB API.
Data queues — This is more a platform feature than a programming language feature. But I add it anyway as it is a great tool for developers. Queues are also natively available on other platforms but rather as in memory data structures (not RPG data structures) to operate in one process and not to communicate between different processes as is often the case on IBM i. So if you have a need to decouple programs but don’t want to install a full blown message queueing system you can often easily achieve things with a data queue (when it comes to one-to-one communications, no pub/sub support). Data queues can also be peristent and really fast.

So these are 12 features of RPG and ILE I really like. There are definitely more great features.

What are your top things you like about RPG and the ILE environment?

Happy coding!

Mihael

Tour of Champions - Update

Sun, 28 Aug 2022 00:00:00 +0200

A little while ago … or even a little while longer ago :) … I made a tutorial about the ILEastic microservice framework : Tour of Champions. It covered making a web service from a template project in a modular style and with SQL for the database access.

And much of my work is to make the life of a developer easier. No matter if it is in the open source world or in a paid environment.

So … can it get easier?

Easier ?

At the time the original tutorial was written you had to compile ILEastic and its dependencies all by yourself. Only the projects Linked List and Message were pre-packaged and could be installed with the iPKG client.

This has changed. Now there are also packages for ILEastic and noxDB2 available which makes it much easier for you to give this tutorial a try and see how easy it is to write web services in RPG without having to switch to any other language (though you could of course use any ILE enabled language in your web service program with ease).

I updated the tutorial so you can have a smooth ride through each section.

The biggest change to the previous setup is that the packaged ILEastic version has been compiled with noxDB2 (which is the UTF8 branch in the Github repository) and is not compatible with the version on Github project site.

Also thanks and kudos to Niels Liisberg for creating these wonderful projects noxDB and ILEastic.

I am very grateful for any feedback.

Happy web servicing!

Mihael

ILEastic ... can it get even easier?

Sun, 24 Apr 2022 00:00:00 +0200

ILEastic is a microservice framework written mostly in C and available as an ILE service program. That means we can build web service in ILE languages like RPG and COBOL (and C of course). No more jumping through hoops to offer a web service to your colleagues or business partners.

Easier ?

Of course we can do/make it easier :) . Because getting ILEastic onto your server has been a pain so far for anyone not familiar with the various projects (namely ILEastic, noxDB and ILEfastCGI) and the tools used for building the projects. Offering a simple save file has not been very easy so far because your system may have a different CCSID as the one used for building the projects which may lead to incompatibilites.

But now I can offer you a nice RPM package which can be installed with iPKG. To install ILEastic with iPKG we just need the iPKG client on our machine (available here) and execute the install command.

1	ipkg install ileastic

But before we can execute this command we need to tell iPKG where it can get the RPM packages from. This can be done with a simple command:

1	ipkg addrepo 'RPGNextGen https://repo.rpgnextgen.com/repository'

This will create the necessary environment for iPKG. By default it will use the library IPKG but you can change that by adding the parameter IPKGLIB(YOUR_LIB).

After telling iPKG where it can find packages we need to tell it which packages can be found at the repository we configured. For this we just need another simple command:

1	ipkg update

Coming back to our original intent of installing ILEastic with the command:

1	ipkg install ileastic

Hmmm … it failed … and that is the correct result. Because ILEastic depends on noxDB we need to install noxDB before installing ILEastic (anybody remembering the old days of rpm where you manually needed to install every required package before installing the package you originally wanted to install?!). Automatically installing all dependencies is definitely on my list but needs much work and won’t be available in the near future. So here we go with first installing noxDB:

1	ipkg install noxdb2

Now we can install ILEastic (see the install command above).

Note that we don’t need to install the noxdb2-devel package. That is the beauty of these packages. You just install what you need. For ILEastic itself we don’t need the prototypes of noxDB. If you want to work with noxDB then of course you can install the noxDB prototypes.

But as we want to use the ILEastic procedures we will need the ILEastic prototypes which are available in the ileastic-devel package. So we are going to install that one too.

1	ipkg install 'ileastic-devel'

iPKG is not a global package manager. It works more like NPM where packages are managed on a per project base (per library in our case). To make a package installation only for a single user you just need to specify a separate library on the commands. To have ILEastic installed in my own library and home directory I would call the following commands:

ipkg addrepo 'RPGNextGen https://repo.rpgnextgen.com/repository' ipkglib(MIHAEL)
ipkg update ipkglib(MIHAEL)
ipkg install ileastic ipkglib(MIHAEL)
ipkg install 'ileastic-devel' ipkglib(MIHAEL) loc('/home/mihael/include')

We can see which packages have been installed by executing a simple command:

1	ipkg list installed

It will display the installed packages and their version.

1
2
3

ileastic (2.0.0)
ileastic-devel (2.0.0)
noxdb2 (2.0.0)

If you want to know more about iPKG take a look at the project wiki and this and this blog post.

And now … how about writing a little web service?!

First Web Service

Starting situation:

noxDB and ILEastic is installed in library MIHAEL (by using iPKG)
NOXDB2 and ILEASTIC service programs are registered in the binding directory MIHAEL/IPKG (automatically done by iPKG)
ILEastic prototypes are installed and located at/home/mihael/include/ileastic/ileastic.rpgle (by using iPKG)
library MIHAEL is part of the library list

Code

We are now set for creating our first RPG web service program with the new ILEastic installation. It will be a FREE RPG program (though we can also use it in a non FREE RPG program but it has to be an ILE program). So our program starts with

**FREE

For this little example we compile it in one go with the command CRTBNDRPG so we can add some compile options in the code (for production I would go with the two step approach of first creating a module with CRTRPGMOD and then creating the program with CRTPGM).

1	ctl-opt dftactgrp(no) actgrp(caller) bnddir('IPKG') thread(*concurrent);

Note: The option thread(*concurrent) is very important because ILEastic is a multi-threaded framework and all our web service programs and service programs are running in the same job but in multiple threads. This compile option makes sure the threads are not tripping on each others toes when it comes to resource and memory management.

Next we need to include the ILEastic prototypes.

1	/include '/home/mihael/include/ileastic/ileastic.rpgle'

I like to directly jump into a procedure at program start without forfeiting the default error handler.

1 2	main(); inlr = on;

And here is our main procedure where we setup the web service and configure our end points / routes.

dcl-proc main;
  dcl-ds config likeds(il_config);

  // The server will listen on port 44000.
  config.port = 44000;
  config.host = '*ANY';

  // Add the end point / route.
  il_addRoute(config : %paddr(sayHello) : IL_GET : '/hello');

  // Starts the server.
  il_listen(config);
end-proc;

Our end point will just return a nice “Hello!” to the requestor.

dcl-proc sayHello;
  dcl-pi *n;
      request  likeds(IL_REQUEST);
      response likeds(IL_RESPONSE);
  end-pi;

  il_responseWrite(response : 'Hello!');
end-proc;

Compile

This program can be compiled like any other FREE RPG program:

1	CRTBNDRPG SRCFILE(MIHAEL/ILEASTIC) SRCMBR(ILHELLO) PGM(MIHAEL/ILHELLO)

As our web service is just a normal ILE program we can start it by simply submitting it with the SBMJOB command.

Start

1	SBMJOB CMD(CALL PGM(ILHELLO)) JOB(ILEASTIC) JOBQ(QUSRNOMAX) CURLIB(MIHAEL) ALWMLTTHD(*YES)

Note: Make sure that the library list is set correctly so that the job has access to all necessary objects.

Now you can query the web service with the tool of your choice like Insomnia, Postman, cURL, … or even your web browser.

The NETSTAT command with the parameter *CNN is sometimes helpful to find the corresponding job because it allows us to select the job by the port it uses.

What’s next?

As ILEastic is now also available via iPKG I might get some time to update the ILEastic tutorial : Tour of Champions. We’ll see :) .

Also thanks and kudos to Niels Liisberg for creating these wonderful projects noxDB and ILEastic.

Happy installing!

Mihael

noxDB ... now even easier

Sat, 26 Feb 2022 00:00:00 +0100

noxDB - Not Only XML - is a software library which makes it easy to parse and generate JSON and XML. Generating JSON or XML directly from an SQL is even easier with noxDB. noxDB has become my favorite tool when working with JSON or XML when I want to process it with a DOM like approach … which is most of the time.

Even Easier

But to top it all off we can make it even easier. Till now it was a bit of a pain to install noxDB (depending on your environment, f. e. installed tools). This has been alleviated as we now have a nice RPM package which can be used with iPKG. To install noxDB with iPKG we just need the iPKG client on our machine (available here) and execute the install command.

1	ipkg install noxdb2

This will install the noxDB service program (NOXDB2) in the library IPKG and also adds it to the binding directory IPKG in the installation library. The installation library can be specified on the install command with the parameter IPKGLIB.

If you want to use the service program you will need the RPG prototypes. These are also packaged and can be installed in the same manner.

1	ipkg install 'noxdb2-devel'

The prototypes are saved in a stream file. By default this stream file will be placed in the IFS in the folder /usr/local/include. But this can also be changed on the install command with the parameter LOC (location).

iPKG is not a global package manager. It works more like NPM where packages are managed on a per project base (per library in our case). To make a package installation only for a single user you just need to specify a separate library on the install command. To have noxDB installed in my own library and home directory I would call the following commands:

1 2	ipkg install noxdb2 ipkglib(MIHAEL) ipkg install 'noxdb2-devel' ipkglib(MIHAEL) loc('/home/mihael/include')

If you want to know more about iPKG take a look at the project wiki and this and this blog post.

Note: The noxdb2 package provides the utf8 branch of the noxDB project which is not yet on par with the main branch when it comes to functionality but covers all the basics already.

Now that we have installed noxDB how to use it in RPG?

Usage

Starting situation:

noxDB installed in library MIHAEL (service program NOXDB2)
NOXDB2 registered in the binding directory MIHAEL/IPKG
library MIHAEL is part of the library list
noxDB2 prototypes installed in folder /home/mihael/include (package noxdb2-devel)
prototypes available at /home/mihael/include/noxdb2/noxDB2.rpgle

We are now set for creating our first RPG program with the new noxDB installation. It will be a FREE RPG program (though we can also use it in a non FREE RPG program but it has to be an ILE program). So our program starts with

**FREE

1	ctl-opt dftactgrp(no) actgrp(caller) bnddir('IPKG');

Next we need to include the noxDB prototypes.

1	/include '/home/mihael/include/noxdb2/noxDB2.rpgle'

I like to directly jump into a procedure at program start without forfeiting the default error handler.

1 2	main(); inlr = on;

And here is our main procedure where we parse a JSON string, modify the noxDB object graph and output it with the DSPLY opcode.

dcl-proc main;
  dcl-s json pointer;
  dcl-s s char(50);

  json = nox_parseString('{ "game" : "Elden Ring" , "releaseDate" : "Q1/2022" }');
  nox_setStr(json : 'releaseDate' : '2022-02-24');
  nox_setStr(json : 'price' : '59,99 €');
  s = nox_asJsonText(json);
  dsply s;

  on-exit;
    nox_close(json);
end-proc;

What’s next?

As noxDB is now available via iPKG providing ILEastic as a RPM package for usage with iPKG is on my list.

Also thanks and kudos to Niels Liisberg for creating these wonderful projects noxDB and ILEastic.

Happy installing!

Mihael

JBatch Example on IBM i - Test Run

Wed, 29 Dec 2021 00:00:00 +0100

This will be the last episode of our Jakarta EE Batch adventure. In this post we will see our example application in action and see how to restart a batch job and the mechanics behind it.

Test Data

To have our application actually do anything we need some test data. I found a nice web site which generates up to 100k data records and you can define how the data should look like. And it is even for free. Great service! Thanks onlinedatagenerator.com.

We can download this test data set from the Bitbucket project site download section and upload it to the IFS on our IBM i server. The data is stored as SQL INSERT statements and also includes the SQL CREATE TABLE statement. We just need a library where we will store the data and we are good to go.

1	RUNSQLSTM SRCSTMF('/home/mihael/customers.sql') COMMIT(*NONE) DFTRDBCOL(RPGNEXTGEN)

Note: The data set contains 100k data records. Depending on your environment you may get some messages in your job log and the job log may even overflow. So you might want to look out for an inquiry message which you need to answer.

Install Application

We can download a precompiled version of the application including all dependencies from the download section of the project site. We can either run it from our PC, IBM i server or any other server. Depending on the host you run the application you will need to adjust the configuration of the database connection.

Run Application

Our application is a Java batch application which can be executed in the same way as any other Java application (and that is one reason I Iike it so much because you don’t have to jump through any hoops to get where you want to go):

Put every jar in the classpath and call the main class.

And we will do exactly that!

1	java -cp batch.jar:libs/* rpgnextgen.batch.Main start customerTransfer --customer 100

We also appended some parameters to the call.

ˋstartˋ : Action parameter (resolved by the class ˋrpgnextgen.batch.BatchStarterˋ
ˋcustomerTransferˋ : Batch job id from the XML file under ˋMETA-INF/batch-jobsˋ
ˋ–customer 100ˋ : These two parameters identify the customer to process. These parameters will get injected into our ˋCustomerReaderˋ class. Optional.

Application Logic

The application either processes one customer (parameter customer) or processes all active customers starting from a given change date (parameter date). If neither customer nor date is provided the application will process all active customers from the current date.

Note: There are no active customers in the given data set. But feel free to modify the data as you want or need.

Note: The date parameter expects the date in ISO format like 2022-01-31 .

Database Configuration

But before we can call our nice little batch program we need to configure our database connection.

Luckily we used MicroProfile Config for our configuration. Now we have multiple ways to get our configuration into the application. We have a base config file in META-INF named microprofile-config.properties. If we don’t do anything else the application will take database configuration from this properties file.

We can override these value by passing Java system properties on the call as additional parameters.

But we can also use environment variables. So under linux we would just execute the following in a shell:

export ibmi_host=localhost
export ibmi_user=mihael
export ibmi_password=uwillneverknow
export ibmi_libraries=RPGNEXTGEN

Note: You will need to adjust those values to match your environment and setup.

Restart Application

The Jakarta EE Batch specification also states that a failed batch run needs to be able to be restarted. For this to work you need to provide some checkpoint information. Checkpoint information can be used in the ItemReader and/or ItemWriter.

And the usage is pretty direct and easy: You just implement the corresponding checkpointInfo methods where you return the checkpoint information for the current state of the reader or writer. It can be any object which can be serialized (implements Serializable).

On a restart of an application the open method is passed the last checkpoint information which we provided in the checkpointInfo method.

@Override
public Serializable checkpointInfo() throws Exception {
	logger.finer("Saving reader checkpoint data: " + lastReturnedCustomer);
	return lastReturnedCustomer == null ? null : lastReturnedCustomer.id;
}

In a real application this could be the last processed customer number (object of class Integer), see class CustomerReader line 64. We would get that Integer on a restart as a parameter on the open method and are checking it at line 43.

if (checkpoint != null) {
	lastCustomerId = (Integer) checkpoint;
	logger.info("Restarting after customer " + lastCustomerId);
}

If we have a lastCustomerId we are starting our processing after that customer by specifying in our SQL the lastCustomerId as our offset to start at.

1
2
3

@SqlQuery("SELECT id, company, street, city, country, discount, active FROM customer "
		+ "WHERE active = 1 AND id > :offset AND changeDate >= :changeDate")
ResultIterator<Customer> listActiveCustomers(@Bind("offset") int offset, @Bind("changeDate") LocalDate changeDate);

But how do we actually restart the application?

For restarting the application we need the last failed batch job id (not IBM i job). We can get this from the log output of the original run. It states:

1	rpgnextgen.batch.BatchStarter: Starting batch job with id 401

So 401 is the batch job id we need to pass to on our restart.

1	java -cp batch.jar:libs/* rpgnextgen.batch.Main restart 401 --date 2022-01-01

Note: It is important to pass the same batch parameters to the restart as on the original batch run. Those parameters are not persisted by the framework.

Wrap Up

I think the batch specification is quite well written and fulfills my needs for batch processing quite well. We can do all the necessary work in a structured way which lets us reuse components and split up workloads in multiple ways and even restart batch runs. Great work!

Happy batching!

Mihael

JBatch Example on IBM i - simplified

Sun, 05 Dec 2021 00:00:00 +0100

The JBatch Example in my last post was ok for a prototype but when you have to write many batch jobs we would rather want to have to code less and have more ready-to-use components. This can be done very easily in only a few steps by putting these reusable components into a batch.core project which we can later declare as a depencency in your batch projects.

Dependencies

As this is about using JBatch as our batch framework of choice for every batch project we can put all those Maven dependencies into the batch.core project. Now the only direct dependency we need to specify in our batch project is batch.core which leaves us with a very clean pom.xml file.

Note: You still need to specify how the application will be build in the build section. If this becomes more complex we could create our own custom packaging type batch in a separate maven plugin project and just specify that plugin in our build section. But that would be overkill at the moment.

Database connection

As we are primarily working with our beloved DB2 on IBM i why not have a database connection readily available for usage?! We just have to create a Jdbi instance provider which takes our database connection configuration via MicroProfile Config, create a DataSource instance and feed that to the Jdbi class which returns a fresh Jdbi instance to be used.

Note: Keep in mind that the Jdbi instance itself is no database connection. We get a database connection by retrieving a Handle instance from the Jdbi instance.

The code may look like this.

@ApplicationScoped
public class IbmiDataSourceProvider {

	@Produces
	@Named("ibmiDataSource")
	public DataSource provideDataSource() {
		Config config = ConfigProvider.getConfig();
		String host = config.getValue("ibmi.host", String.class);
		String user = config.getValue("ibmi.user", String.class);
		String password = config.getValue("ibmi.password", String.class);
		String libraries = config.getValue("ibmi.libraries", String.class);

		AS400JDBCDataSource ds = new AS400JDBCDataSource(host, user, password);
		ds.setLibraries(libraries);
		return ds;

	}
}

This code only supports a single database connection configuration (which is my use case) but can be extended to support multiple database connections to different machines.

Note: This will be even more compact when SmallRye Config switches to the jakarta namespace and we can use CDI to access the configuration.

Note: Helidon has an extension which builds an injectable DataSource instance purely from configuration. Very slick and developer friendly! :)

@ApplicationScoped
public class IbmiDatabaseProvider {

	@Inject
	@Named("ibmiDataSource")
	DataSource dataSource;

	private Jdbi jdbi;
	
	@PostConstruct
	public void postConstruct() {
		jdbi = Jdbi.create(dataSources.get());
		jdbi.installPlugin(new SqlObjectPlugin());
	}

	@Produces
	public Jdbi provideJdbi() {
		return jdbi;
	}
}

Utility Classes

The following utility classes will also make it into our new batch.core project.

BatchExecutorServiceProvider
ConsoleWriter
EndJobListener
PassThroughItemProcessor

Application Starter

The classes Main and BatchStarter can also be moved to the batch.core project as they have no project specific code. Main is our entry point for the application and needs to be specified on application start like this:

java -cp batch.project.jar:libs/* batch.core.Main start batchJobId

The Rest

This just leaves us with the batch project specific classes like the implementation of ItemReader, ItemProcessor, ItemWriter and database access classes like Jdbi DAO interfaces, mapper and data models.

That are just 4 classes for the example project with just 152 lines of code. We just cut our lines of code to a third of the original code base. I think that is not a bad result.

jbatch-example/src/main/java/rpgnextgen/batch/data$ find . -name '*.java' | xargs wc -l
  36 ./Customer.java
  25 ./CustomerMapper.java
  17 ./CustomerDao.java
  74 ./CustomerReader.java
 152 insgesamt

And when it comes to configuration … as we are using MicroProfile Config we don’t need to configure anything in the batch project itself as most configuration is done in the batch.core project. We just need to override the ibmi.user and ibmi.password variable conveniently via environment variables on the target system. Very easy to do.

Note: We probably can even do without any configuration at all because the IBM Toolbox for Java supports using the user of the current job. AFAIK you would need to specify localhost for the server and *CURRENT for the user. But for some reason that didn’t work. If anybody has come up with the solution for this I would be very happy for a message on how to do it.

Happy simplifying!

Mihael

JBatch Example on IBM i

Sun, 28 Nov 2021 00:00:00 +0100

My last blog post was about batch processing on IBM i with Jakarta EE Batch implementation JBatch from IBM. I didn’t get into too much detail in that post as it was only to provide a brief overview of Jakarta EE Batch. In this blog post I want to present and explain one of my JBatch example applications available on Bitbucket.

Setup

For using JBatch you don’t have to do very much. You can use JBatch either with an Jakarta EE application server or even stand alone. I would like to use it standalone so I need to include all the necessary Jakarta EE API projects in my application, f. e. JAX-B API, Transactions API, CDI API. And more or less that would be it. JBatch needs to save some batch runtime data somewhere. I will be using Apache Derby for that.

Note: With pure JBatch you can get away with about 4 direct dependencies and all in all about 10 resolved (including transitive) dependencies. Now compare that to any Node.js application which has a bit more functionality than a “hello world” example. That is one reason I find the Java world so much more compelling than the Node.js world.

I will also use the MicroProfile Config implementation SmallRye Config. Sadly SmallRye Config still uses the javax namespace instead of the new jakarta namespace. But that will probably change in the future. So we will need to include the javax-API dependency to our project and cannot use the ConfigExtension to integrate it with CDI.

To get all those moving parts work together I will be using JBoss Weld.

For accessing the database I will use the Jdbi project. Jdbi eases the usage of SQL in Java without getting in your way. The result is a very small and clean code base for your database access layer. Kudos to the Jdbi project.

And of course we need a database driver and as we access our beloved IBM i server we will use the JTOpen project.

All in all I end up with having 14 direct dependencies and with 45 resolved dependencies. Again … compare that to a Node.js batch application. Actually I am currently working on another open source Node.js application and ls -1 node_modules | wc -l counts 169 libraries.

Now let’s take a look at the actual code.

Start Application

The application starts in the batch.Main class. First we will start the CDI container and initialize it:

1
2
3

weld = new Weld().containerId("rpgnextgen.batch");

weld.initialize();

More or less that’s it for this class.

Start Batch Job

Our application needs to wait till the CDI container is ready. It will fire a ContainerInitialized event when it is ready. The batch.BatchStarter class listens for this event to actually start the batch job.

1
2
3

public void startImport(@Observes ContainerInitialized event, @Named("commandLineArguments") List<String> parameters) {
      ...
}

If the event is received it parses the command line arguments to see what action to take and what batch job to start or restart.

start customerTransfer --customer 100 --date 2021-01-31

Also every additional parameter pair is added to the batch properties which are passed to the batch job. Those batch properties can be injected into the batch classes like this:

1
2
3

@Inject
@BatchProperty
private String customer;

With the information we can now actually start the batch job like this.

private void runBatch(String batchName, Properties batchProperties) {
      BatchExecutorServiceProvider executorServiceProvider = new BatchExecutorServiceProvider();

      BatchSPIManager batchSPIManager = BatchSPIManager.getInstance();
      batchSPIManager.registerPlatformMode(BatchSPIManager.PlatformMode.SE);
      batchSPIManager.registerExecutorServiceProvider(executorServiceProvider);

      long jobExecId = BatchRuntime.getJobOperator().start(batchName, batchProperties);
      logger.info("Starting batch job with id " + jobExecId);
}

Restart Batch Job

To restart a batch job the job execution id is needed which is outputted to the log on starting the batch job. The command line arguments look very similar to the previous ones.

restart <job exec id> --customer 100 --date 2021-01-31

The batch properties appended as parameters to the command line arguments need to be passed again.

Instead of calling the JobOperator::start method it calls the JobOperator::restart method.

BatchRuntime.getJobOperator().restart(execId, batchProperties);

Step Execution

Jakarta EE Batch supports the Input → Processing → Output type of flow with the provided interface ItemReader, ItemProcessor and ItemWriter but it also supports a much simpler and task-oriented style (not coupled to any input items) with Batchlet.

Our declared steps for this job look like this:

<step id="step1">
      <chunk item-count="100">
            <reader ref="rpgnextgen.batch.data.CustomerReader">
                  <properties>
                        <property name="customer" value="#{jobParameters['customer']}" />
                        <property name="date" value="#{jobParameters['date']}" />
                  </properties>
            </reader>
            <processor ref="rpgnextgen.batch.util.PassThroughItemProcessor" />
            <writer ref="rpgnextgen.batch.util.ConsoleWriter" />
      </chunk>
</step>

Data is read from the database in the CustomerReader class. The class will get the customer and date batch property injected (if passed).

All read items from the reader are just passed through by the PassThroughItemProcessor (that would be the place where we would do any validation or transformation).

And all processed items are outputted to the console by the ConsoleWriter.

Reading Data

The CustomerReader retrieves the data from the database. For convenience it extends AbstractItemReader which is provided by JBatch. The open method is where the initialization gets done like getting a database connection (Jdbi Handle instance) and in our case we also execute the SQL query already in the open method but don’t retrieve any items from the result set yet.

The actual readItem method is rather short and we even can get away with a one-liner. Our DAO object returned an iterator from the SQL query. We can now just query the iterator for the next customer.

1	return lastReturnedCustomer = activeCustomers.hasNext() ? activeCustomers.next() : null;

Note: The last returned customer is temporarily saved for later use when it comes to check points and restarting the batch job.

The close method is all about finishing and cleaning up like closing any database connections.

Jdbi

For database access Jdbi is used in this project. Jdbi’s API comes in two flavors: Fluent API and SQL objects.

This project uses SQL objects from Jdbi. This is a pretty slick way of doing database queries. The SQL query is just defined as an annotation and Jdbi is doing the heavy lifting.

It can look as simple as this:

1 2	@SqlQuery("SELECT id, company, street, city, country, discount, active FROM customer WHERE id = :id") ResultIterator<Customer> getCustomer(@Bind("id") int id);

For mapping the raw SQL data to a Customer object a Jdbi RowMapper is used which just takes the values from the result set and maps it to a Customer object.

@Override
public Customer map(ResultSet rs, StatementContext ctx) throws SQLException {
	Customer c = new Customer();
	c.active = rs.getInt("active") == 1;
	c.city = rs.getString("city");
	c.country = rs.getString("country");
	c.discount = rs.getBigDecimal("discount");
	c.id = rs.getInt("id");
	c.company = rs.getString("company");
	c.street = rs.getString("street");

	return c;
}

Could this have been coded with less lines of code by using some “magic”? Yes, definitely. But I like simple code even if it means some lines more to write and this mapping can be understood by any programmer from beginner to ninja.

ItemProcessor

As the project uses the Input → Processing → Output processing style of JBatch an ItemProcessor needs to be implemented. In this example I don’t want to do anything with the item but just passing it down the pipe to the output like this:

1
2
3

public Object processItem(Object item) throws Exception {
	return item;
}

Writing Data

To make things simple I just write the customer data to the console using the toString method of the object. The ConsoleWriter is easy to write and useful in many cases where you just want to test some workflows.

public void writeItems(List<Object> items) throws Exception {
	items.forEach((c) -> {
		System.out.println(c);
		count++;
	});
}

As a little bonus ConsoleWriter outputs the number of processed items at the end of processing in the ItemWriter::close method.

As we have only this one execution step that is our whole batch workflow.

Dependencies

In this blog post I mentioned the number of dependencies compared to a Node.js application. RPG would probably win in a dependency contest. But that is also one problem in RPG: You don’t have so many open source projects to choose from and you end up with less functionality out-of-the-box compared to other programming languages.

When I take a look at the libraries I pulled into this project I notice that almost all of them come from very reliable sources like IBM, Red Hat, Eclipse and Apache. I have almost no library where I cannot immediately associate it with a reputable and reliable source. Again … compare that to a Node.js application.

Wrap Up

It may look like very much code for very little functionality but with this setup we have everything neatly seperated and in small chunks of code which is much easier to grasp than those RPG monoliths which are thousands of lines long.

And if we look at the lines of code per source file there are suprisingly few lines in each source file. Not even one source file with more than 100 lines of code. Each source file fits on one screen which makes it very easy to understand.

$ find . -name '*.java' | xargs wc -l
32 ./rpgnextgen/batch/jdbi/IbmiDataSourceProvider.java
40 ./rpgnextgen/batch/jdbi/IbmiDatabaseProvider.java
36 ./rpgnextgen/batch/data/Customer.java
25 ./rpgnextgen/batch/data/CustomerMapper.java
17 ./rpgnextgen/batch/data/CustomerDao.java
74 ./rpgnextgen/batch/data/CustomerReader.java
89 ./rpgnextgen/batch/BatchStarter.java
70 ./rpgnextgen/batch/Main.java
28 ./rpgnextgen/batch/util/ConsoleWriter.java
19 ./rpgnextgen/batch/util/EndJobListener.java
12 ./rpgnextgen/batch/util/PassThroughItemProcessor.java
23 ./rpgnextgen/batch/util/BatchExecutorServiceProvider.java

And even if this seems like much code and many classes we can pull many of these classes into a separate reusable Maven project and only need to write the code specific to our task. But this will be covered in the next blog post. :)

Happy batching!

Mihael

JBatch on IBM i

Thu, 18 Nov 2021 00:00:00 +0100

This whole blog has been very RPG centric so far. But some of you also know that I also do some Java programming (and also some Node.js+TypeScript from time to time). Lately I got involved into some Java projects which also include IBM i as a potential target platform.

Java on IBM i has a long history … not always with the best results. Java performance on IBM i was not very good at the start but has improved drastically so that you probably don’t have to worry too much about any differences in runtime to other platforms. But that may depend on your requirements. If every microsecond counts you might be very picky about frameworks, the JRE and the platform it runs on.

My opinion on that is that Java performance on IBM i for the SMB market is more than sufficient. But as IBM i is not only used in the midrange market but also in the high end of computing side-by-side to main frames things may look different in these situations.

But let us stay in the SMB market.

Java on IBM i

The Java Runtime Environment OpenJ9 can be installed via yum which will give you a JRE 11 on IBM i. Just recently the new long term support release of Java version 17 has been released. Hopefully Java 17 will soon come to the IBM i platform so that all the new nifty features like the new Record type can be used.

Batch Processing on IBM i

The batch processing on IBM i was always the domain of RPG and CL. Java didn’t have any say in it so far. But I think with the latest Jakarta EE Batch specification to have the same feature set in RPG or CL you would have to do some serious development work.

Some of those features are:

splitting up work into multiple threads (either splitting up the items or the work flow)
reusing and mixing of predefined steps
declaratively defining the workflow (in XML)
restart jobs on failure at last failed transaction
job history (runtime, completion status, …)
listener interfaces (equivalent to exit points)

It is not that those features are not doable in RPG or CL. In fact I think they are all doable in RPG but they are not available by default and most RPG programs have not been written with those features in mind.

Jakarta EE Batch + DIY

I have searched for some time for a good framework for batch processing in Java but have not found a package or product which supports everything I envisioned for my batch application. So I will have to put the pieces together by myself.

Jakarta EE Batch looks like a good spec for batch processing in Java and JBatch implements it nicely. The good thing is that I can use all the common Jakarta EE annotations and don’t have to learn a completely new framework or approach (like Spring) because I already got enough on my plate. So using the already widely used and supported approach and style is very convenient for me. Jakarta EE Batch also covers all the batch features I need.

Note: Yes, Spring Batch has influenced the development of Jakarta EE Batch a lot but it is not the same. Jakarta EE Batch is also influence by some other frameworks and platforms.

I could have just sticked to JBatch and be done. But there are so many ways and frameworks which make my life easier like CDI, MicroProfile Config, etc … which I also wanted to use. As I haven’t found any prepackaged solution I just package it myself. So it is

JBatch (Jakarta EE Batch) + Weld (CDI) + SmallRye Config (MP Config) + JDBI (database access)

Results

With this stack it is not only easier to write batch programs in Java but also has a pretty decent performance. One test case was to process a data set of 500k records from one table and update about 25k records (record by record) in another table.

This was executed in a breeze by the RPG program in about 4 seconds. But the Java program did also not bad at all with 6 seconds (not taking into account the ramp-up time of the JVM and of the frameworks). You can argue now that the Java program took 1.5 times longer to process the data but the table layout favored more the RPG style of execution as the data set to be processed could not be selected by any index because of the way the data was stored (f. e. dates stored as numbers in the format ddmmyyyy :( ).

So in different cases Java might actually be faster than RPG. But execution time is not always the most important factor. If some approach is fast enough other criteria might be much more interesting, f. e. low maintenance, available developers, active community, fast SDLC, future safety, … just to name a few.

Cons

Some things might be implemented in an easier way. For example the creation of the data source is done manually. This might be done automatically with just some configuration entries, like in Helidon.

Also the dependency injection is not (fully) working in any class which is directly part of the JBatch process, like any ItemReader or ItemWriter as these are not created by Weld but by JBatch and thus don’t fully support CDI yet.

Helidon

Helidon is a micro service framework from Oracle. It comes with everything you need in the micro service world. But it can also be used as a one-stop-shop for all those rudimentry things like CDI, data sources, etc. you need or are useful in a batch program. Helidon 2.x does not support the new jakarta namespace but sticks to the javax namespace for the java enterprise packages. So it does not work with IBM JBatch 2.x because starting with the 2.0.0 version IBM switched to the jakarta namespace.

But you can use Helidon with the JBatch 1.0.3 version. To get all the good stuff of Helidon you would probably need to use helidon-microprofile-cdi as a parent Maven artifact in your project. That artifact has most things included but the web server which you probably don’t need in your batch program. The result is a stripped down Helidon server which you can extend by adding Helidon extensions.

Apache BatchEE

The Jakarta EE Batch implementation placed under the Apache Geronimo project also looks very promising and comes with some additional features like a management UI for running batch processes. But the project does not seem to be very active and so I am sticking with JBatch from IBM.

Example

Actually I wanted to include an example application in this post but this one is already long enough as it is so the JBatch example application be featured in another post.

If somebody is successfully using any other Java based solution I would gladly like to here about it.

Happy batching!

Mihael

International Domain Names and RPG

Fri, 12 Nov 2021 00:00:00 +0100

I just realized that I haven’t written any new blog entry for some months now. It is not that there was nothing interesting to write about but most things were either not a topic for the blog or was too complex/big for the blog.

This blog entry is about IDN or IDNA - International Domain Names (in Applications) - and Punycode.

Punycode … ? WTF is Punycode?

Punycode is an encoding algorithm to encode Unicode strings to ASCII strings and vice versa in a lossless way. Non-ASCII characters will be removed from the string and appended to the string in an encoded form in only ASCII characters. The result is an ASCII string which can be decoded to the original Unicode string.

Use Case - Domain Name Encoding

For almost two decades domain names can contain non-ASCII characters (for example german Umlaute). This has been slowly implemented and supported by the software vendors and projects. In 2003 Punycode was introduced as a major building block for Internationalizing Domain Names in Applications (IDNA) to provide a way to convert domain names with unicode characters to an ASCII string and vice versa.

One may think that today after almost two decades there is no further need for such things as Punycode … but that is far from the current state of affairs. Even today you may need to encode a domain name with Punycode into an ACE (ASCII Compatible Encoding) string which is a Punycode encoded string. If the original string contains non-ASCII characters it is prefixed with “xn–”. Also all characters are converted to lowercase.

But not only in DNS server configuration / maintenance will you encounter ACE and Punycode. You may also encounter it everywhere where you use a domain name, f. e. at such a common thing as an e-mail address.

RPG Support

After a short while of searching the internet for a library which may be ported to either RPG or ILE C I found the projects libidn and its successor libidn2. These two libraries have also some bindings for other languages. But nothing for RPG so far.

But after some more searching I have found a real good article about IDNA using a Punycode implementation from the ICU project.

ICU … that should ring a bell. ICU (International Components for Unicode) is an open source project which is sponsored and supported by IBM. The implementations are primarily C/C++ and Java. But there are also ICU packages for many other programming languages. And after searching ICU on IBM i Docs I was surprised to find many references to ICU and also an RPG example using a function from the ICU library. Actually ICU is installed in the library QICU and provided as ILE service programs which means it is also usable in an RPG program. QICU also contains the binding directory QXICUAPIBD which has all latest ICU service programs registered.

ICU Punycode - Unicode to ASCII

So far this looked really promising but my enthusiasm was quickly diminishing. Because besides the mentioning of ICU I have found no further detailed information on how to use these available service programs in my RPG program.

@IBM: Where is the API documentation and RPG prototypes?

It seems there is none … or at least not readily available. So forget the easy way. Lets do it the hard way. I took a closer look at the service programs and have found an exported function u_strToPunycode_4_0 which sounds exactly what I was searching for:

Converting a unicode string to an ASCII string with Punycode.

Ok. I had the name of the function but what about the parameters and return value? After browsing through the source code of the ICU project I have found a header file: punycode.h

/**
 * u_strToPunycode() converts Unicode to Punycode.
 *
 * The input string must not contain single, unpaired surrogates.
 * The output will be represented as an array of ASCII code points.
 *
 * The output string is NUL-terminated according to normal ICU
 * string output rules.
 *
 * @param src Input Unicode string.
 *            This function handles a limited amount of code points
 *            (the limit is >=64).
 *            U_INDEX_OUTOFBOUNDS_ERROR is set if the limit is exceeded.
 * @param srcLength Number of UChars in src, or -1 if NUL-terminated.
 * @param dest Output Punycode array.
 * @param destCapacity Size of dest.
 * @param caseFlags Vector of boolean values, one per input UChar,
 *                  indicating that the corresponding character is to be
 *                  marked for the decoder optionally
 *                  uppercasing (TRUE) or lowercasing (FALSE)
 *                  the character.
 *                  ASCII characters are output directly in the case as marked.
 *                  Flags corresponding to trail surrogates are ignored.
 *                  If caseFlags==NULL then input characters are not
 *                  case-mapped.
 * @param pErrorCode ICU in/out error code parameter.
 *                   U_INVALID_CHAR_FOUND if src contains
 *                   unmatched single surrogates.
 *                   U_INDEX_OUTOFBOUNDS_ERROR if src contains
 *                   too many code points.
 * @return Number of ASCII characters in puny.
 *
 * @see u_strFromPunycode
 */
U_CFUNC int32_t
u_strToPunycode(const UChar *src, int32_t srcLength,
                UChar *dest, int32_t destCapacity,
                const UBool *caseFlags,
                UErrorCode *pErrorCode);

That looks much more promising. Now to decode all the parts of the function header.

1  U_CFUNC int32_t
2  u_strToPunycode(
3                  const UChar *src, 
4                  int32_t srcLength,
5                  UChar *dest, 
6                  int32_t destCapacity,
7                  const UBool *caseFlags,
8                  UErrorCode *pErrorCode);

The return value is an int32_t which is a 32-bit integer which translates to int(10) in RPG. It represents the length of the encoded string (number of characters).
The function name does not match the exported name 100%. The name is u_strToPunycode and the exported name has the version appended: u_strToPunycode_4_0. This can be easily solved by setting the exported name in the extproc keyword on the prototype.

Now comes the more tricky part.

In the documention we can see that the strings are expected to be in UTF-16 character encoding, see ICU Guidelines. So the passed string variable needs to have the data type ucs2 and CCSID UTF-16. We also pass the variable by reference (as a pointer).
The function either expects to get the size of the source string passed as the second parameter or get -1 passed to indicate that the source string is terminated with a hex null value (x’00’). The length is passed to the function by value.
The destination string which will contain the encoded string needs to be passed by reference (as a pointer) so that the function can write the encoded content to the variable.
The size of the destination variable is passed as an integer by value.
The function accepts hints about the case of the seperate characters. This is passed as an array of UBool values. UBool translates to an int(10) in RPG. But lucky for us we can just pass *NULL indicating that the case should not be mapped.
The last parameter is an error indicator. The ICU API declares and uses the UErrorCode data type for the result of functions. UErrorCode translates to int(10) in RPG and holds 0 if successful. Anything else is an error. The variable needs to be initialized with 0 before calling any ICU functions, see error handling in the ICU Guidelines.

With this information we can create the following prototype:

dcl-pr u_strToPunycode int(10) extproc(*CWIDEN : 'u_strToPunycode_4_0');
  source pointer value;
  sourceLength int(10) value;
  dest pointer value;
  destLength int(10) value;
  caseFlags pointer const;
  errorCode int(10);
end-pr;

If you put all the puzzle pieces together the result may look like this:

**FREE

ctl-opt dftactgrp(*no) actgrp(*caller) bnddir('QICU/QXICUAPIBD');

// U_CFUNC int32_t
// u_strToPunycode(const UChar *src, int32_t srcLength,
//                 UChar *dest, int32_t destCapacity,
//                 const UBool *caseFlags,
//                 UErrorCode *pErrorCode);
dcl-pr u_strToPunycode int(10) extproc(*CWIDEN : 'u_strToPunycode_4_0');
  source pointer value;
  sourceLength int(10) value;
  dest pointer value;
  destLength int(10) value;
  caseFlags pointer const;
  errorCode int(10);
end-pr;


main();
*inlr = *on;


dcl-proc main;
  dcl-s domain ucs2(50) ccsid(*utf16);
  dcl-s encoded ucs2(50) ccsid(*utf16);
  dcl-s rc int(10);
  dcl-s errorCode int(10);

  domain = 'Küchen.de' + x'00';
  
  rc = u_strToPunycode(%addr(domain) : -1 : %addr(encoded) : %size(encoded) : *null : errorCode);
  if (errorCode = 0);
    dsply 'Success';
  else;
    dsply 'Nope!';
  endif;
end-proc;

The variable encoded now has the Punycode encoded string: kchen.de-65a

Note: This is just the Punycode encoded value, not an ACE string. xn–kchen.de-65a would be the corresponding ACE string.

But hey! I am just an application developer. I have nothing to do with IDNs.

Yes, that might be. But f. e. sometimes you have to work with e-mail addresses and those contain domain names. Your e-mail or mailing list provider may expect ACE strings as domain names and reject non-ASCII characters in domain names.

This concludes our adventure with Punycode, IDNA and ICU.

Happy punycoding!

Mihael

noxDB API Documentation

Fri, 27 Aug 2021 00:00:00 +0200

I am using noxDB for some time now and I always stumbled upon some API calls where I just couldn’t remember either the exact function name, parameters or return value.

Now it is not only all readily available in my ILEDocs instance at rpgnextgen.com for looking up the information but also for every editor which supports ILEDocs comments like MiWorkplace.

This makes the life of the modern RPG developer so much easier.

For everyone else you can still look up the API in the RPG copybook as the ILEDocs comments are side-by-side with the prototypes, constants and templates or take a look at the Sitemule tutorials at github.io.

Happy mouse-hovering and CTRL+SPACEing!

Mihael

Kong Dashboard

Tue, 17 Aug 2021 00:00:00 +0200

Kong Dashboard

Kong (or Kong Gateway) is a very lightweight API gateway. What is an API gateway?

An API gateway takes all API calls from clients, then routes them to the appropriate microservice with request routing, composition, and protocol translation. Typically it handles a request by invoking multiple microservices and aggregating the results, to determine the best path. It can translate between web protocols and web‑unfriendly protocols that are used internally. — NGINX

It usually acts as a single entry point for API requests into your system. This makes it easy to configure and implement cross-cutting concerns like authentication and logging at a single point.

Why a UI project?

Kong Gateway is not distributed with a UI. So configuring Kong Gateway is done on the command line (which is not a bad thing per se). It also does not provide a simple CLI but needs to be configured by executing HTTP requests. This makes it not very user-friendly to configure and maintain.

But on the other hand having web services to use for configuration makes it so much easier to write a frontend by myself.

So here we go … Kong Dashboard!

Installation

I will not cover installing Kong. This has been done dozens of times. The website at konghq.com is a good place to start. Note: Do not use the DB-less mode as this mode cannot be used with the Admin API.

Kong Gateway listens on two ports.

Port for every normal API request
Port for Kong Gateway Admin API

You configure both in the Kong Gateway configuration file. By default the Admin API is not exposed to the outside world. It is only reachable on localhost. This makes it somewhat secure as anybody who wants to access the Admin API needs to login to the server hosting the Kong Gateway.

Self hosting the Admin API

But with this setup we cannot reach the Admin API with our web application.

One solution is to let Kong Gateway host the Admin API itself by adding a route

and a service for the Admin API.

Securing the Admin API

Ok. But wait … now the Admin API is open for everybody to use.

Yes. That is correct. But as it is now hosted by Kong Gateway we can use all the features of Kong Gateway to make it more secure, f. e. configuring the JWT plugin for the Admin API route.

First we need to configure a consumer. Take a look at the Kong Gateway documentation on how to add a consumer.

Then we need to configure the JWT plugin at the just created consumer. The documentation for the JWT plugin is quite good and worth reading. If you are using JWT with the algorithm HS256 you just need to pass a key and a secret like

{
    "key" : "my_key_which_identifies_token_generator",
    "secret" : "my_secret_which_is_used_for_token_verification"
}

The key is the link between the configured JWT plugin in Kong Gateway and the incoming JWT token. Kong Gateway searches for the configured key in the incoming JWT token. By default it searches for the iss attribute in the JWT header and payload part. The attribute can be configured in the JWT plugin configuration, see key_claim_name. It uses the secret from the first matching JWT plugin configuration for token verification.

We also want to check the expiration date of the token. This is not done by default and must be configured. We can PATCH the configuration by sending

1	{ "config" : { "claims_to_verify" : ["exp"] } }

Last but not least we need to activate the plugin. We have configured the plugin but it will not be used yet. We can activate it in our defined route kong-admin by POSTing the following to /routes/kong-admin/plugins

1	{ "name" : "jwt" }

Note: You also need to enable the CORS plugin either globally or for this route.

Web Application

The Kong Dashboard is an Angular web application and does not need an extra web service layer. It communicates directly with the Kong Admin API. We can take the transpiled Angular web application and drop it on any web server.

If you are using a secured Admin API you can configure Kong Dashboard to ask for a JWT token which will be added to the HTTP requests going to the Admin API, see assets/config.json.

{
    "kong" : [
        {
            "name" : "Kong Node 1",
            "url" : http://kong-node1/kong/api
        }
    ],
    "auth" : {
        "enabled" : true,
        "protocol" : "oauth2",
        "grant_type" : "client_credentials",
        "url" : http://oauth2_provider/api/oauth
    },
    "help": https://github.com/m1h43l
}

Note: Only OAuth2 grant type client_credentials is supported at the moment.

Wrap Up

So this is my new open source project Kong Dashboard.

Is it feature-complete?

Definitely not! But it covers the basics like routes, services, consumers and a dashboard. My next task on the project will be to implement plugin configuration support.

Is is production stable?

We have used it for some months now and it worked fine for now. So I would give it at least a beta version status.

It would be great to get some feedback. What works for you and what doesn’t? What are you missing the most in this new Kong UI project?

Happy routing!

Mihael

ILEastic Tutorial - SQL Transactions

Sun, 13 Jun 2021 00:00:00 +0200

In most IBM i shops SQL is the primary way of accessing data or is at least on its way to be it. And this is a good thing. But as ILEastic is a multi threaded framework we need to be aware of the fact that multiple threads may be using SQL transactions simultaneously. We need to make sure that these threads don’t step on each others toes when it comes to transactions.

IBM has a wonderful solution for this where you actually have to do almost nothing.
The answer is ( *drumroll* ): SQL Server Mode

From the IBM i Documentation web site:

There are cases where it is desirable for transactional work to be scoped to the thread, rather than an activation group. In other words, each thread has its own commitment definition and transactional work for each commitment definition is independent of work performed in other threads.

Db2® for i provides this support by using the Change Job (QWTCHGJB) API to change the job to run in SQL server mode. When an SQL connection is requested in SQL server mode, it is routed to a separate job. All subsequent SQL operations that are performed for that connection are also routed to that job.

So we just have to call the Change Job API at the beginning of our web service program and we are ready for thread scoped transactions. Great!

Sometimes it is a bit tricky to call some system APIs. How does it look like with the Change Job API?

Not problem at all. We can easily encapsulate the code in a procedure and call that procedure at the start of the web service program.

dcl-proc setSqlServerMode;
  dcl-pr sys_changeJob extpgm('QWTCHGJB');
    job char(26) const;
    internalJobId char(16) const;
    format char(8) const;
    jobChangeInfo char(1000) const options(*varsize);
    errorCode likeds(qusec);
  end-pr;

  dcl-ds jobc0200 qualified template;
    numberKeys int(10) inz(1);
    fieldInfoLength int(10) inz(20);
    keyField int(10) inz(1922);
    dataType char(1) inz('C');
    reserved2 char(3);
    lengthData int(10) inz(1);
    data char(1) inz('1');
    reserved3 char(3);
  end-ds;


  dcl-ds error likeds(qusec) inz;
  dcl-ds jobData likeds(jobc0200) inz(*likeds);

  sys_changeJob('*' : '' : 'JOBC0200' : jobData : error);
end-proc;

Information about the Change Job API can be found at IBM i Documentation. There is also a Technote in the Redbooks section from Scott Forstie called DB2 for i5/OS: SQL Mode Primer (published in 2007). As we can see from the naming of the operating system this book is not brand new but there are many things we can learn from it.

One important fact that I didn’t find on the new IBM i Documentation web site is which job attributes are passed from the main job to the SQL Server Mode job. Luckily this can be found in the previously mentioned Technote.

The following attributes are taken from the user profile: …

A very important thing I missed on my first read of that Technote is that most attributes are taken from the user profile of the the main job and not the job attributes of the running job itself. So if you change the attributes (like the current library) in your program or on submitting your job it really doesn’t matter in regard to the SQL Server Mode job because the SQL Server Mode job will use the current library from the user profile and not from the running job.

The SQL Server Mode job also uses the default library list (system and user part). So we need to make sure that every table we want to access via SQL in a non-qualified way (without specifying the library) is in the default library list configured in the system or in the current library of the user profile.

Note: One last thing to mention is that I have an unresolved problem with the SQL MERGE statement when run in SQL Server Mode. In my case changing the MERGE statement to an INSERT and UPDATE statement worked fine.

You can find an example on how to use this in the Tour of Champions demo project and you can also find this information in the Web Service section of the tutorial.

If you have an idea what topic I should also cover in this series just drop me a mail.

Happy microserving!

Mihael

STOMP Reloaded

Sun, 06 Jun 2021 00:00:00 +0200

Most people have probably assumed that the next blog post will again be about “ILEastic Tutorial : Tour of Champions”. I will definitely continue this series but I had some time to work on the STOMP client again. And this time I have working STOMP service program which implements the STOMP specification 1.2.

Background info: Some years ago I started my journey with developing an ILE STOMP client for communicating with the world outside of IBM i, specifically with message queueing systems like Artemis ActiveMQ and RabbitMQ. Back then the current STOMP specification was version 1.0. Today we have the specification version 1.2 and a couple of features have been added like content types and heart-beating.

But why even use an extra message queueing system? We already have native data queues on IBM i!

Yes, we have data queues. And it is a great tool which is very easy to use. Thanks IBM (really :-) )! But you need to use the right tool for the right job. With data queues we need to specify upfront the maximum message size. This can have a drastic impact on the maximum number of entries for a data queue. And you always have to work around the fact that the queue can be full. What to do with any extra messages?

Another thing is that you can “only” do a 1:1 relationship between publisher and consumer when using data queues. You can add multiple consumer to a data queue but a message will only be sent to one consumer, not to all. This means you cannot do something like pub/sub, 1:n relationship where you publish a message and each subscribed consumer gets the same message.

Message Encoding

With the specification version 1.0 we were tied to sending messages in ASCII character encoding. Today we can send the data in more or less any encoding supported by the platform. The implementation of the client allows to set the default character encoding with the following line:

1	stomp_setContentType(client : 'application/json;charset=utf-8');

If nothing else is passed to the stomp_command_send procedure everything will be encoded with the previously set default encoding. If we have a message which should be encoded differently we can specify a different MIME type (and character set) on the stomp_command_send procedure.

1	stomp_command_send(client : '/queue/ticker' : '{ "id" : 358 }' : *omit : 'application/json;charset=utf-16');

You don’t have to worry about how to convert the data to the desired character set. This is automatically done by the client. You just pass the data encoded in the CCSID of the current job and set the desired character set (added as a character set attribute of the MIME Type).

The STOMP Client project comes with some predefined constants for the most often used MIME types, see MIME types in the ILEDocs documentation.

A list of supported character sets and their mapping to a CCSID is available in the wiki of the project.

Binary Data

Even though STOMP is a text oriented protocol with the new specification you can also send binary very easily.

dcl-s data pointer;

data = loadBinaryData();

stomp_command_send(client : '/queue/binaries' : data : *omit : STOMP_MIME_BINARY);

This assumes that the data is null-terminated. If you want to send data which may include a x’00’ value you need to set the message data length (4th parameter) like this:

dcl-s data pointer;
dcl-c DATA_SIZE 1024;

data = loadBinaryData(DATA_SIZE);

stomp_command_send(client : '/queue/binaries' : data : DATA_SIZE : STOMP_MIME_BINARY);

Logging

The first version of the STOMP client used the great Log4RPG package from Thomas Raddatz for getting optional output for debugging purposes. This worked very well and was very flexible. But it also introduced some problems when it comes to deployment. Log4RPG depends on the two service programs from tools400.de, IFS and Basics. Among those projects Log4RPG and Basics are rather heavy weight and offer a great deal of functionality which we don’t use in the STOMP client project. We just need to output some messages to the job log for debugging purposes. So now we got an extremely simplified logging which can be activated by setting the environment variable STOMP_DEBUG to 1.

ADDENVVAR STOMP_DEBUG 1

Note: The environment variable is queried at the initialization of the service program. A service program is only loaded once in the activation group. If you want to change the logging you need to start the client in a new activation group.

Installation

The STOMP projects relies on three other projects: Linked List, Message and libtree. We could build the dependencies first and then build the STOMP client itself but this is very tedious.

Luckily there is a much easier solution: iPKG

I have already built all dependencies and the STOMP client and created iPKG compatible packages. So we can just use the package manager and install everything with some simple commands on the 5250 command line.

To install the dependencies just execute the following commands:

1
2
3

ipkg install linkedlist
ipkg install message
ipkg install libtree

To install the STOMP client just execute the following command:

1	ipkg install stomp

We also need the STOMP copybooks for the prototypes, constants and templates. These are stored as stream files. By default these will be placed into /usr/local/include. We can change that by specifying a different location on the ipkg command:

1	ipkg install 'stomp-devel' loc('/home/mihael/include')

Much easier than compiling everything yourself.

Examples

The source code repository has an examples folder which contains some basic examples on how to use the STOMP service program.

Wrap Up

There is a ton of additional features these message queuing systems have beyond a simple data queue. And having an additional system doesn’t mean it cannot run on IBM i. ActiveMQ Classic and ActiveMQ Artemis (the successor to the classic version) are both written in Java and can easily run on IBM i.

It would be great if you could post some feedback. I am very interested which message queueing system you are using the STOMP client with and if it worked out of the box (some projects don’t adhere so strictly to the spec as others).

And what about speed?! In some small tests I was able to send 1000 messages from one client in 215 milliseconds. I think that is not such a bad value for starters. How fast is your system?

Happy stomping!

Mihael

ILEastic Tutorial - Thread Local Storage

Sat, 15 May 2021 00:00:00 +0200

Another part of the mini series “ILEastic Tutorial : Tour of Champions” is available: Thread Local Storage.

This part will cover what Thread Local Storage is and how it can be used. It will open up a whole new load of possibilities from authentication to authorization and more specific setting of CORS headers.

You can find it in the Web Service section of the tutorial.

If you have an idea what topic I should also cover in this series just drop me a mail.

Happy microserving!

Mihael

ILEastic Tutorial : Plugins - Route Id

Thu, 13 May 2021 00:00:00 +0200

This is a little update on the mini series “ILEastic Tutorial : Tour of Champions”. Last time I covered plugins, those “special” procedures which are executed on the exit points of the ILEastic request handling workflow.

There is another little feature which is also part of the plugins chapter of the tutorial: Route Id.

We can add an id to the registration of a route / end point. By doing this we can identify the route ILEastic is using for the request from inside a plugin. This allows us to further remove code for cross-cutting concerns from the end points and put them centrally into the plugins.

Example

We want to set different CORS headers for each route and not just add “allow all” HTTP headers to the response.

How do we know which end point is used for the request?

Very simple! Just query the subfield routeId of the il_request data structure. It will contain the string we have passed on the il_addRoute call (last parameter).

il_addRoute(config : %paddr(list) : IL_GET : '/api/champion' : *omit : 'champion_list');

...
dcl-proc list;
    dcl-pi *n ind;
        request  likeds(IL_REQUEST);
        response likeds(IL_RESPONSE);
    end-pi;

    ...

    if (request.routeId = 'champion_list');
        ...
    endif;

    ...
end-proc;

Now we can differentiate between the different end points in the plugin procedures.

You can also find this information in the Web Service section of the tutorial.

If you have an idea what topic I should also cover in this series just drop me a mail.

Happy microserving!

Mihael

ILEastic Tutorial - Plugins

Sat, 01 May 2021 00:00:00 +0200

This is the second part of my mini series “ILEastic Tutorial : Tour of Champions”. Today I will cover ILEastic plugins. In Java you would call them “Servlet Filter”. They intercept a request before any routing to an endpoint/servlet has been done. It is a great way to extend the functionality of your service.

You can find it in the Web Service section of the tutorial.

If you have an idea what topic I should also cover in this series just drop me a mail.

Happy microserving!

Mihael

ILEastic Tutorial - Tour of Champions

Sun, 25 Apr 2021 00:00:00 +0200

This will be a series of little articles which show how to implement a microservice project on IBM i. I will start from a blank (or template) project and work my way through the different parts of the application. I will be using tools like git, bash, ssh, make to cover my basic needs for a development environment. As we are on IBM i I will use the native microservice framework ILEastic for the web services, my favorite RPG editor MiWorkplace ;-) for editing RPG and DBeaver for all stuff related to SQL.

The content of the series is already available on the new website https://champions.rpgnextgen.com.

Each part of the series will be appended to the content of this website. It currently covers the basics like setting it all up, database, service programs and the first web service. Later parts of the series will cover things like SQL transactions in a multi threaded environment, authentication (Basic Auth and OAuth 2.0), embedding a template engine, error handling, administration, configuration provisioning and load balancing. Perhaps you have some other topics related to this. Please let me know, perhaps I can also cover that.

I hope you enjoy the ride and realize that RPG is one of the best options for web services on IBM i because you have access to the whole IBM i platform natively and directly.

Keep it simple! Use your mind!

Happy microserving!

Mihael

ILEDash - First Alpha Release

Sat, 03 Apr 2021 00:00:00 +0200

I have been developing web services with ILEastic for some time now and the framework is really great in what it does and provides. It handles all the HTTP protocol stuff for you. It supports things like a plugin system, Basic Auth, JWT, JSON, request routing out of the box with a developer friendly API (documented with ILEDocs and available at iledocs.rpgnextgen.com. As a framework it really does not have to hide behind those microservice frameworks from other programming languages.

What it does not provide is a good way to manage those web services. And it doesn’t have to. Its main focus is microservices via HTTP.

In a previous post I already mentioned how one could manage ILEastic web services with native IBM i tools. For a couple of services this will probably suffice. But if you have dozens of web services you can easily lose the overview.

This is the moment where you need some software which helps you to manage your services and provides you with a good overview over your services.

ILEDash to the rescue!

ILEDash is a new project from me and myself. You specify your web services in the web application of ILEDash and not just the programs itself but you can also specify the configuration for each service. You stop and start a service with just a button click. Running multiple instances of a web service is also no problem if you need to cover a higher volume of requests (f. e. in combination with a load balancer like the Apache HTTP server on IBM i which I will cover in another post).

Technology

The backend is powered by native RPG programs using the ILEastic microservice framework. The frontend uses Angular for providing a clean and functional UI for managing your services.

And because of this technology mix you can run it all on your IBM i. No other servers needed. ILEastic is native ILE and the web frontend can be hosted with the Apache HTTP Server on your IBM i server.

Open Source

There is currently so much buzz about Open Source on IBM i and I never hear anything about the funtastic and fantastic Open Source ILE projects. Perhaps this project will get the ball rolling, meaning this project is Open Source.

Open Alpha

The project is currently in its alpha stage. Meaning it generally works but there are probably some edge cases which need some more attention. Everybody who wants to give it a try is welcome. Please send me a mail at mihael@rpgnextgen.com if you are interested and you can find the software in the download section on the website. You can find the installation instructions in the installation section on the website.

Documentation

This is one thing the projects needs to massively improve because the documentation is basically non existent. This will be one of my next tasks to work on.

But for starters you really just need to start the subsystem. The necessary jobs will be started automatically as they are configured as autostart jobs in the subsystem description.

If you need any help just contact me.

Feedback

Currently most features I need are already implemented. It would be great to get some feedback on your experience with ILEDash and your thoughts on what is good or bad about the project and what you are missing.

I am looking forward to hear from you all.

Happy Easter Day!

Mihael

Error Handling

Sun, 28 Feb 2021 00:00:00 +0100

I think this is a difficult topic and many people will obviously have their own solution to this problem. And this is not really an RPG only topic. This problem exists in every programming language (I have encountered so far).

What to do when an error can occur?

Should I catch/monitor the error? Should I log the error? Should I do anything?

And especially the last question is the one where most people will say: “Of course you need to do something!” But this is not true at all. If you cannot handle the error in the location you are (in your program or stack) then you probably should not do anything and let the error “bubble up” to the next level in the call stack where it might be handled in a better way.

Log and Throw

This is a common thing we have learned is a “NoNo”. But this is where our IBM i platform (especially the ILE environment) is different than other platforms:

Everything is already logged in the job log!

Everything? Probably not. But it is enough to say that we don’t need any additional logging strategy for errors. That does not mean that we never need any additional logging at all. More log statements may better show the flow inside the program and other log frameworks may use another medium beside the job log. But that is another topic.

But “Log and throw” cannot be excluded on IBM i as every escape message is automatically logged in the job log and we may want to communicate to the caller a specific message or condition. So we may rethrow the caught escape message with a specific message from our own message file.

Sending an Escape Message

Sending a message from a message file does not have to be complicated. It can be as easy as calling a single procedure.

1	message_file_escape(*omit : 'DMOMSGFILE' : 'DMO0010');

Now the caller can use a monitor block to catch that escape message and analyze it.

/include psds.rpginc
/include qusec.rpginc

...

dcl-ds errorCode likeds(qusec) inz;

...

monitor;
  demo_persist(data);
on-error;
  if (exception = 'DMO0010')
    sendValidationErrorResponse();
  else;
    // cannot handle any other error here => send it to the next call stack entry
    message_resendEscapeMessage('' : errorCode);
  endif;
endmon;

You can find the procedures about messages in the message project and the API documentation at my ILEDocs site.

Escape Messages only for real Errors

When you want to communicate something back to the caller which is not an error then don’t use an escape message. It is way more effecient to use something like a constant as a return value than an escape message to do that. Handling escape messages is much more expensive than just checking a value.

Documentation

And document what the user can expect! If a program or procedure sends an escape message in a certain situation then write it down so that the user of the program or procedure can handle it if possible (or send it further up the stack).

The same goes for web service endpoints: document what the web client can expect!

Happy error handling!

Mihael

Message Service Program

Sat, 27 Feb 2021 00:00:00 +0100

The message project has been extended by some procedures. Now you can also use messages from message files.

You can get the message service program from the save file in the download section of the project or fetch the source from the git repository. The documentation has been uploaded to my ILEDocs site.

This also enables us to more easily communicate specific errors to the caller. How to do this and where it can be very useful will be displayed in another blog post.

Happy messaging!

Mihael

Modernization

Fri, 01 Jan 2021 00:00:00 +0100

I think there is not one topic in the IBM i community which has been talked so much about as modernization. And here I am also talking about it :) .

The modernization drum has been beaten for as long as I can think of and every time I hear about it it is just about putting lipstick on a pig and moving things to the web with the least effort possible no matter what the result looks like or what the underlying programs still look like. But I won’t handle the frontend or web topic here.

From the top of my head I only remember a single article where they have used a staged approach with first bringing it all to the web with a screen converting product but then also doing the next step and doing a full rework of the whole application. Most people are only doing step one but missing the really important step two.

But this post is about the backend accessing the database and the first thing I hear is

Use SQL views! Abstract the database tables away and use views!

In some cases it probably makes sense. BUT NOT IN ALL!!! Come on! Use your mind! If the database table has already names that make sense, if there is no additional logic to be captured in views … if there are no not used columns or misused columns … (fill in some more conditions) … then why should I use a view?!?!

What I also hear and read a lot is

Make a database access layer!

In my opinion this is something which has been swept into the IBM i community from other platforms where you don’t have such a good database access method like we have in RPG where database access is built into from the start. On other platforms you don’t have native database support at all. On these platforms database access is much more complex and it all comes with additional software libraries, keyword ORM. Some might say now

Hey, but doesn’t f. e. Java has JDBC and JPA?

Yes. That is right. But that is something which doesn’t come with the stock JRE (ok … JDBC does come with standard JRE). The JPA interfaces come with the JEE Runtime and then you still have no implementation to work with. (I won’t cover JDBC here because this is not something you would like to work with in the first place when using Java and like to access the database). For JPA you need a JPA implementation (like EclipseLink or Apache OpenJPA) and additionally the database driver so how much built-in is it really? I don’t mean that it doesn’t work. I had developed and maintained a project for seven years which used JPA on an x86 based platform and it worked great (also much thanks to OSGi, which will soon move under the Eclipse umbrella , and Apache Karaf) but my opinion is that it is not as much built-in as the database support is built into the RPG language.

So coming back to the database layer … I don’t think that you need an additional layer in RPG that does nothing else but database access. So rather than having like two or sometimes three layers for justing getting the data to the next level (database layer, business layer, transformatiton layer for transforming from internal model to external model) I like to put everything in a single layer. I don’t say that you never need any additional layer but only a single one … but use your mind. Start with the simple approach and look how far it takes you and at what point it doesn’t suffice. And how much more does it cost to have a more complex approach and is it worth it? What extra value does an extra layer gives you which you really need?

The simple approach for me would be to optional make SQL views if needed or just plainly use the database tables. Don’t blindly make SQL indexes. Use visual explain to check where you need an index and where the database engine doesn’t use any index you created in good faith. Start with one layer to access the database and make your data available to any layer which presents the data to the frontend. Put a data structure for the data and necessary prototoypes in a copybook to let the next layer interface with the data access layer. Don’t let any database specifics trickle through to the next layer. By this it doesn’t matter if the next layer is a 5250 program, any other service program or a web service done with ILEastic. And by doing this it doesn’t matter if some or all of the data comes from the local database or if some or all of it is pulled from some other (web) service.

It does not matter! … and it shouldn’t!

So if you really have read the article this far you probably already got the message I try to bring to you with this post

Don’t always generalize!!! Use your mind!

Don’t make a database access layer just for the case of having such a layer especially if it doesn’t make anything but a straight read and write without doing anything else. For some data I don’t even write any layer but let the last layer access the database directly because the data doesn’t need any additional management and is extremely simple and rather static and in some cases also read-only. In this case the database itself is the interface to the data. Is this the best way for all cases? NO!!! But you need to decide that from case to case. There is no one-size-fits-all!

So the overall message is

Use your mind!

Have a great new year!

Mihael

Demonstrate My Style

Thu, 31 Dec 2020 00:00:00 +0100

There are many web sites which discuss the technical parts of RPG covering the latest bifs and opcodes and they are great for looking up specific things about RPG.

But when you are new to RPG and perhaps programming in general you actually might not know what a “good” programming style is and you may have not discovered or developed your own coding style.

So here I want to give you something to start with : Demonstrate My Style.

This is not something written in stone and not a general rule which you have to follow but my personal preferences at the time of writing which you may use as a starting point to develop your own style.

For me coding is something between craftsmanship and art. It is not just about performance and business values. And having something which just barely works is not something I pursue.

Happy coding!

Mihael

Thread Local Storage and ILEastic

Sat, 05 Dec 2020 00:00:00 +0100

Thread Local Storage (TLS) is a widely used programming method that uses static or global memory local to a thread. — Thread Local Storage on Wikipedia.

So TLS is all about having a storage area per thread. This is not much of a concern for most RPG developers. Most of us are working with jobs and not directly with threads. But if you take a look at the details of a job (DSPJOB OPTION(*THREAD)) you will see that your job has one thread, the main thread. And all your programs are executed in this main thread.

So what good is TLS for you? It can help you even if you have only your main thread. You can put something in that storage and at a later time you can retrieve it. This is really helpful if you have something you need to save at some point in your programs (f. e. at login) and don’t need it 99% of the time. But you don’t want to pass this data around as a parameter to all programs or procedures. So putting that in the TLS and retrieving it 10 entries deeper in your callstack can make things easier.

But many readers will now say “Hey, we already got that covered. We got LDA and GDA for this.”. Yes, you can use LDA for this and it is a nice for storing data at the scope of the job. But how much data can fit into the LDA? Not very much. And how often did you step onto the toes of another application storing data in the same LDA position, f. e. 3rd party software?

Implementation

There are many ways how to implement TLS. A very easy implementation can be done in RPG. Yes … your eyes don’t betray you. It is good old RPG. And we even need only a single control option for this:

thread(*concurrent)

This will make your RPG module thread safe. From the IBM Knowledge Center:

If THREAD(*CONCURRENT) is specified, then multiple threads can run in the module at the same time. By default, all the static storage in the module will be in thread-local storage, meaning that each thread will have its own copy of the static variables in the module, including compiler-internal variables. This allows multiple threads to run the procedures within the module at the same time and be completely independent of each other.

You don’t have to do any extra. Just put that control option into your module. Then declare a global variable and more or less you are already done. You can write some procedures to manage access to this variable. But that’s it!

“But how is this better than using LDA?”

For example you can use this in a multi-threaded and in a single-threaded environment. It doesn’t matter. LDA will not work in a multi-threaded environment or should I rather say it will have some side effects as each thread accesses the same data area.

And as you implement it yourself it is totally up to you what features your TLS has. F. e. you can use a noxDB graph as your TLS. This means you can even store hierarchical data. Or have one graph branch for each part of your application, f. e. /user for all your user data in the job. So you would query /user/name for the name of the user and /user/roles to get an array of roles the user has or /user/email for his email address. No more toe stepping. It is totally up to the implementation.

You can find a simple implementation here on Bitbucket.org.

“But doesn’t a new service program instance gets created with each new activation group?”

Yes. But if you specify a named activation group for this service program it will always be executed in the same named activation group and everything should be fine.

TLS and ILEastic

ILEastic is a microservice framework for creating web services in ILE languages, f. e. in RPG. Yes, your reading is just fine. No PASE or Java involved. Just some great ILE modules bound together in a nice service program. ILEastic creates a new thread for each request (as most web and application servers do today).

So this is where our thread local storage comes into play.

You can do some initialization of your TLS in a plugin which is registered on PREREQUEST.

il_addPlugin(config : %paddr('web_util_tlsPlugin') : IL_PREREQUEST);

dcl-proc web_util_tlsPlugin export;
  dcl-pi *n ind;
    request  likeds(IL_REQUEST);
    response likeds(IL_RESPONSE);
  end-pi;

  dcl-s threadLocal pointer;
  dcl-s username varchar(100);
  dcl-s tls pointer;

  threadLocal = il_getThreadMem(request);
  
  // The ILEastic JWT plugin places the payload of the JWT under /ileastic/jwt/payload.
  // We don't have to worry about not having a value here because without a proper 
  // token or payload the request is returned with a 401 Unauthorized response and 
  // never reaches this line.
  username = jx_getStr(threadLocal : '/ileastic/jwt/payload/username' : 'unknown');

  tls = tls_getStorage();
  jx_setStr(tls : '/user/name' : username);

  return *on;
end-proc;

Means it will get executed before the request is handled by our end point / route procedure. If you need to do some cleanup you can register a plugin on POSTRESPONSE.

il_addPlugin(config : %paddr('web_util_tlsFree') : IL_POSTRESPONSE);

dcl-proc web_util_tlsFree export;
  dcl-pi *n ind;
    request  likeds(IL_REQUEST);
    response likeds(IL_RESPONSE);
  end-pi;

  tls_freeStorage();

  return *on;
end-proc;

ILEastic has its own TLS. You can access it by calling il_getThreadMem(). You get a noxDB graph which you can use as you wish. Only the branch /ileastic is reserved for ILEastic itself.

But you don’t want to have ILEastic as a dependency for every program or service program. Especially not those which have nothing to do with web services. So having ones own TLS is a much cleaner solution.

Note: When registering plugins be aware that they will be called by ILEastic in the same order you are registering them.

Use Case

A good use case for this would be the following:

You have created a web service with ILEastic.
This web service uses a JWT token for authentication.
ILEastic checks the token and puts the token data into its thread local storage by using the ILEastic JWT plugin.
You use your own TLS plugin to store that token data (like user id, name, …) in your own TLS.
Down the callstack you access your TLS to store in the database which user has created or changed that database entry.

For example you can use this in a multi-threaded and in a single-threaded environment.

As I have said you can use the same TLS service program in your 5250 interactive and batch programs. You just need to initialize your TLS with the data from the current job. In our use case this would be the user name from the current job. So if you now query /user/name you still get a valid value, regardless of the environment.

Happy threading!

Mihael

Working with Validation Lists

Sun, 11 Oct 2020 00:00:00 +0200

Probably not many of you have heard of validation lists on IBM i. So from the IBM Knowledge Center

Validation list objects provide a method for applications to securely store user authentication information.

Validation lists can store any data associated with a key. The good thing about a validation list is that you have the option to have the data one-way or two-way encrypted which makes it an ideal option for storying authentication data. One other reason for using validation lists instead of user profiles is that you don’t have to create a whole user profile on the system when you only want to authenticate a “user”.

IBM has delivered a great option for authenticating “users” but failed on delivering all tools needed for easily working with validation lists. It has provided several APIs for accessing the validation list but failed on providing higher level service programs and commands.

But as the API is documented in the IBM Knowledge Center the IBM i community has all the information needed for creating those tools by themselves. And Carsten Flensburg did so and provides all the necessary code and information in his really good “APIs by Example” series and on his website.

MANY THANKS!!!

The only thing you now need to do is download the code and compile it on your machine. That can be a hurdle for one or the other and thus I have packaged all the validation list commands, panel groups and other necessary objects into one iPKG package for easy installation. Now you can install all the validation list tools from Carsten Flensburg with a single command

ipkg install vldl

Can it be any easier?! No … i think not (at least not for 3rd party software). You can find the packaged code on the project web site. The iPKG packages can be found at https://repo.rpgnextgen.com.

Happy validating!

Mihael

Web Services Setup with ILEastic

Sun, 04 Oct 2020 00:00:00 +0200

I have written some articles about writing web services with ILEastic and it is really as simple as already shown. And most of the time it is not any easier in other languages. You have to do/check/consider the same things regardless of the programming language.

But what I didn’t cover so far was how to deploy and run the web service once we have finished developing it. Deployment is really easy and it doesn’t take a lot to cover that because with ILEastic your web service program will be a native IBM i object in the QSYS file system. You only need to put the compiled object in the library where you want to have your own objects.

Setting up the web service to run when the system starts is another thing. And there are probably many ways to skin the cat. I want to show you a simple one (not necessarily the best one).

Subsystem

I typically want to have my web services all in a separate subsystem. So we create a brand new subsystem for them.

CRTSBSD SBSD(MY_OBJ/WS) POOLS((1 *BASE)) TEXT('Web Services')

That only creates a description of the subsystem. You can start it with the STRSBS command.

Class

This class has nothing to do with a programming construct or concept. A class defines how many resources a job can get, like processor time, memory and number of threads. Remember that ILEastic is a multi-threaded web framework so we need to be able to create multiple threads in a job.

CRTCLS CLS(MY_OBJ/WS) TEXT('Web Services')

Job Queue

As jobs are entering a subsystem through a job queue we will need to create at least one job queue and attach that to the subsystem. You can attach more than one job queue to a subsystem.

CRTJOBQ JOBQ(MY_OBJ/WS) TEXT('Web Services')

We need yet to attach the job queue to the subsystem.

ADDJOBQE SBSD(MY_OBJ/WS) JOBQ(MY_OBJ/WS) MAXACT(*NOMAX)

Job Description

The job description defines the environment of a job during its runtime in the system. Things like printer, logs, library list, multi-threading capability, which user profile is used for running the job and also which program is called initially at job start.

CRTJOBD JOBD(MY_OBJ/WS_HELLO) JOBQ(MY_OBJ/WS) USER(WS) RQSDTA('CALL PGM(HELLOWS)') 
    INLLIBL(MY_OBJ SHARED ILEASTIC) ALWMLTTHD(*YES) TEXT('Web Services : Hello World')

Every job runs under a certain user profile. If it is an interactive job then the user has logged himself into the system. A submitted job uses the same user profile as the original job which has done the submit. But if a job is autostarted when the subsystem is started there is no job from where to inherit the user profile. So you need to specify a user profile in the job description and don’t use the default value *RQD.

Note: When you are already using iPKG for installing 3rd party open source software you will probably want to add the IPKG library to the library list of the job description.

Autostart Job

The web service should automatically start when the subsystem is started. This can be achieved with autostart jobs. You need to specify which job description should be used with the autostart job. This also determines which web service (program) is started on job start, see request data of the job description.

ADDAJE SBSD(MY_OBJ/WS) JOB(WS_HELLO) JOBD(MY_OBJ/WS_HELLO)

We can see now that we need a job description for every web service we want to automatically start at subsystem start.

Routing Entry

If you now start your new subsystem you will see that there is nothing to see. No jobs are running in your subsystem. That is because every job is routed internally in the subsystem to a program which executes the request data of the job (from the job description). Depending on the routing you can specify that a job in the subsystem has different resource limits by assigning a certain class object (remember the previously created class object).

The request data is just text for the subsystem on which to route to a specifiy program with certain resource limits (class).

To make things easy we use the QCMD program to excute our request data (which is a simple CALL) and use the previously create class object for every started job.

ADDRTGE SBSD(MY_OBJ/WS) SEQNBR(9999) CMPVAL(*ANY) PGM(QCMD) CLS(MY_OBJ/WS) MAXACT(100)

If no routing entry matches the request data of a job then the job is just dropped. I have yet to figure out where to find any log entry about the dropped job.

Restart Web Services

Restarting a web service is really simple because you just need to kill the old job submit a new one with the corresponding job description and that’s it.

SBMJOB JOBD(MY_OBJ/WS_HELLO) USER(*JOBD) RQSDTA(*JOBD) CURLIB(*CRTDFT) INLLIBL(*JOBD)

Now you can start your subsystem and have your web service say hello to the world! :)

Happy setting up!

Mihael

Dynamic Configuration with ILEastic

Wed, 22 Jul 2020 00:00:00 +0200

My last post was about sharing configuration in an ILEastic web service. This is especially handy when you split your web service application into multiple modules. This is not necessary but may be a good idea if you have many end points / routes so that your modules don’t grow too big.

But sometimes the configuration is dynamic and you can’t put it manually into a table which is later read from the web service application.

Use case: You want to dynamically start new web service instances depending on the utilization. So beforehand you don’t know how many instances/jobs you will start and each instance needs at least another port.

Solution: You define a port range. You provide a configuration web service which among other attributes contains a free port number from the port range. All provisioned configurations are persisted somehow/somewhere so that you know which ports are already in use.

So we don’t push the configuration to the web service instance f. e. by passing it as a program parameter (which is not easy or even possible if you have a big configuration structure) but we let the web service ask for the configuration when it is ready to do so.

How will the web service know where to ask for the configuration?

This can be achieved by setting an environment variable to the url the web service instance will query. Keep in mind that the url also needs to be dynamic and unique so that no other web service instance fetches the “wrong” configuration.

Note: Don’t forget to add the CPYENVVAR(*YES) parameter to the submit job command.

So the code for loading the configuration in the web service may look something like this:

dcl-proc loadConfig;
  dcl-pi *n pointer end-pi;

  dcl-pr getenv pointer extproc('getenv');
    name pointer value options(*string:*trim);
  end-pr;

  dcl-s config pointer;
  dcl-s data varchar(1000);
  dcl-s url varchar(1000);
  dcl-s sqlString varchar(10000);

  // get the configuration url from the environment variable
  url = %str(getenv('CONFIG_URL'));
  if (url = *blank);
    message_escape('No config url provided.');
  endif;

  // get the configuration (in json format) from the previously retrieved url
  sqlString = 'SELECT * FROM ( Values( SYSTOOLS.HTTPGETCLOB( ? ,'''') ) )';
  EXEC SQL PREPARE statement FROM :sqlString;
  EXEC SQL DECLARE c CURSOR FOR statement;
  EXEC SQL OPEN c USING :url;
  EXEC SQL FETCH FROM c INTO :data;
  if (sqlcode = 0);
    config = json_parseString(data);
  endif;

  return config;
  
  on-exit;
    EXEC SQL CLOSE c;
end-proc;

Note: This is only a prototype! Please add error handling where needed and keep in mind that this will start a Java VM for every job because of the usage of the SYSTOOLS HTTP SQL function.

Happy provisioning!

Mihael

Sharing Configuration in ILEastic

Sat, 20 Jun 2020 00:00:00 +0200

I stumbled upon a little problem I first had some trouble to wrap my head around. My situation was the following:

I had an ILEastic web service with multiple modules
I wanted to share the configuration data with all modules

So for this kind of internal sharing of data I had decided to use a global variable I exported in one module and imported in the other module.

So for my first module the code looked something like this:

**FREE

ctl-opt thread(*concurrent);

/include 'ileastic/ileastic.rpgle'
/include 'noxdb/jsonparser.rpgle'

dcl-s userConfig pointer export;


main();
*inlr = *on;


dcl-proc main;
  dcl-ds config likeds(il_config);

  userConfig = loadConfig();

  ...
end-proc;

And the second module with the code for the web service endpoints looked something like this:

**FREE

ctl-opt nomain thread(*CONCURRENT);

/include 'ileastic/ileastic.rpgle'

dcl-s userConfig pointer import;

...

Everything looked nice. The procedure loadConfig did load the configuration and the configuration was assigned to the variable userConfig. All seemed well but when the procedure in the second module accessed the imported variable the variable hold no value.

Hmmm …

After much reading in the IBM Knowledge Center I haven’t found anything which made it work.

And I noticed that I find most of my solutions on my way home from work or before going to bed. And that was also the case here.

I hadn’t taken into account that this was a multi-threaded application and with thread(*CONCURRENT) I have a new userConfig variable for every thread I started. So the value I previously set is not available in the thread I started. But I wanted the configuration to be available in all threads so a little change to the code made it happen.

1	dcl-s userConfig pointer export static(*allthread);

and

1	dcl-s userConfig pointer import static(*allthread);

Now with the keywords static(*ALLTHREAD) there is only one instance of this variable for all threads.

Next post will be about provisioning the web service with a dynamic configuration.

Happy configuring!

Mihael

I can package it all by myself

Thu, 21 May 2020 00:00:00 +0200

… and also provide those packages to others.

But you probably know this expression much better as “I can read it all by myself”. I have read those books from Dr. Seuss very often and with pleasure. Some were real tongue twisters.

But what I really like about the books is that they inspire you to do things yourself. To not wait for others to do the work. And this is a great problem in the IBM i community as many are waiting for IBM to do the work. But when we look at other platforms it is not the big companies who started the contributions to their community. It is the small developer in his spare time. Getting done what he always wanted to have or needed.

And this is what the iPKG project is about. We need some easy way to build, package and deploy our projects and iPKG can be a part of that.

This post is about the packaging of software.

The ipkgbuilder project (sister project of iPKG) creates RPM packages which are compatible with the iPKG client (but not with yum).

It is written in Java and needs at least a Java 1.8 Runtime Environment (JRE). It can run either on the PC or on the IBM i server.

The source will be available at the project site on Bitbucket.

There is already a precompiled binary in the download section at the project site.

Connection Properties

The connection.properties file contains the following entries:

server
user
password
buildlibrary

If you are running the ipkgbuilder on the IBM i server you can drop the entries server, user and password as it uses the current job for the connection.

Spec File

The spec file is a well formed XML file which contains everything about the content of the RPM file.

The structure of the XML is pretty flat. It has a top element package with a couple of sub elements.

name : package name
version : package version
buildVersion : build version
group : category of the package, see list of categories/groups
summary : summary of the package content (one liner)
description : more detailed description of the content
license : license of the software
url : URL of the software (like project website)
vendor : vendor or author of the software
packager : name of the packager (which may be the same as vendor but don’t have to)
osVersion : Version of the IBM i operating system
requirement : dependencies of the package
provides : specifies what the package provides besides itself (see virtual packages
dataPackage : mark package as a data package
directoryPrefix : The directory prefix specified the part of the file path which can be replaced on installation, see relocation.
exportedModule : see iPKG documentation about IPKG binding directory
exportedServiceProgram : same as exportedModule
file : absolute file path of each packaged file or object
purgePrefix : remove beginning part of the file path
preInstallScript : QCMDEXC script, see iPKG documentation about scripts
postInstallScript : QCMDEXC script, see iPKG documentation about scripts
preUninstallScript : QCMDEXC script, see iPKG documentation about scripts
postUninstallScript : QCMDEXC script, see iPKG documentation about scripts

At my spec files repository you can see the spec files used for the repository at RPG Next Gen.

Example spec file of the stream project:

<?xml version="1.0" ?>
<package>
	<name>stream</name>
	<version>1.0.0</version>
	<buildVersion>1</buildVersion>
	<group>Development/Library</group>
	<summary>Streaming API for ILE</summary>
	<description>A stream consist of an emitter which provides the data for the stream. The data can be piped through pipes and may end in a sink.</description>
	<license>MIT</license>
	<url>https://bitbucket.org/m1hael/stream/</url>
	<vendor>Mihael Schmidt</vendor>
	<packager>Mihael Schmidt</packager>
	<osVersion>7.3</osVersion>
	<requirement>arraylist &gt;= 2.0.0</requirement>
	<requirement>arraylist &lt; 3.0.0</requirement>
	<requirement>message &gt;= 1.1.0</requirement>
	<requirement>message &lt; 2.0.0</requirement>
	<dataPackage>false</dataPackage>
	<directoryPrefix>/QSYS.LIB/MIHAEL.LIB</directoryPrefix>
	<exportedServiceProgram>/QSYS.LIB/MIHAEL.LIB/STREAM.SRVPGM</exportedServiceProgram>
	<file>/QSYS.LIB/MIHAEL.LIB/STREAM.SRVPGM</file>
	<purgePrefix></purgePrefix>
</package>

Package Version

The version is best specified in the semantic versioning scheme, like x.y.z where x is the major version number, y is the minor version number and z is the revision number. The version can also be specified as x.y or just x. Only numbers are allowed in the version number. No other characters and no version qualifiers.

Build Version

The buildVersion is the version of the package used by the packager to identify differences in packages to previous versions of the package. The original content of the package may be the same but the packager may have patched it and thus the package needs to be marked different from the previous package. The build version consist of a single number.

OS Version

The osVersion is also specified using the semantic versioning scheme, f. e. 7.3 .

Requirements

Each requirement is specified in a new requirement element. See the iPKG documentation what can be specified as a requirement and how.

Data Package

A data package is treated as the other packages. The only difference is the handling of the location where the content is placed. The objects will be restored in the data library specified on the command, see parameter DTALIB. Valid values are true and false.

Build

There is a ready to use jar file in the download section at the project site. The program needs the following parameters:

connection.properties : Properties files which contains the connection settings and the build library
spec file : XML file which contains the definition of the RPM file
output directory : Directory for the RPM file

Example:

java -jar ipkgbuilder.jar connection.properties my_project.spec packages_directory

Note: The name of the RPM file is determined by the attributes in the spec file, like name, version, etc… .

QSYS.LIB

Objects from the QSYS.LIB file system are packed into a save file in the build library. The save file is the payload of the RPM package.

IFS

All other stream files are packed into a zip file in the /tmp folder. The zip file is the payload of the RPM package.

Check RPM Entries

You can check the entries of the RPM file with the embedded redline library.

java -cp ipkgbuilder.jar org.redline_rpm.Scanner my_project-1.0.0-ppc-ibmi-7.3.rpm

The Scanner program outputs all RPM header entries (and chokes on the payload).

Repository

One of the reasons for choosing the RPM format was the existing tools available on various platforms to generate and process RPM packages.

One of those tools is createrepo which takes a set of packages and generates some XML files about them. Those XML are processed by the iPKG client to get some information about the repository and the packages hosted in that repository.

Note: createrepo is also available on IBM i via yum.

Wrap it all together

A build script for a repository might look like this:

#!/bin/bash

rm -rf repository/repodata

SPECFILES=$(ls specfiles)

for SPECFILE in $SPECFILES
do
    echo "Building from spec file $SPECFILE ..."
    java -jar ipkgbuilder.jar connection.properties specfiles/$SPECFILE repository/
done

createrepo --no-database --content ibmi --distro rpgnextgen -p ./repository/ -s md5

# ipkg doesn't process gzip files yet => need to unzip
gzip -k -d ./repository/repodata/*.gz

# provide a convenient zip file for offline installation
rm -f rpgnextgen-repository.*
zip -r rpgnextgen-repository.zip repository
md5sum -b rpgnextgen-repository.zip > rpgnextgen-repository.md5

# upload all to hosting platform
ssh rpgnextgen.com@ssh.rpgnextgen.com "rm -rf /www/repo/repository/*"
scp -r repository/* rpgnextgen.com@ssh.rpgnextgen.com:/www/repo/repository/
scp -r repository/repodata/*primary.xml rpgnextgen.com@ssh.rpgnextgen.com:/www/repo/repository/repodata/primary.xml
scp rpgnextgen-repository.* rpgnextgen.com@ssh.rpgnextgen.com:/www/repo/repository/

# purge cache on hosting platform
ssh rpgnextgen.com@ssh.rpgnextgen.com "cache-purge https://repo.rpgnextgen.com"

What’s next?

Now we can easily script the build of RPM packages. The next level of package creation automation would be the integration into a build server like Jenkins. Jenkins uses pipelines to define the build script. So having a pipeline command for creating an RPM package would be great.

What do you think? Please send me your feedback!

Happy packaging!

Mihael

Streaming API for ILE

Fri, 08 May 2020 00:00:00 +0200

For those of you who cannot associate anything with streams in programming:

A stream consists of an emitter (provider) of data and a sequence of one or more consumers of data whereas the consumer may pass the data further to the next consumer constructing a streaming data pipeline.

Emitter

The emitter can produce a fixed or infinite number of data blocks. A stream is not restricted to one type of data. The emitter produces events which just hold a pointer to the data. Behind the pointer can lie any kind of data.

Consumer

Consumer of data can be categorized in pipes and sinks.

A pipe has multiple action choices:

process the data and push it to the next consumer
change the data but keep the type and push it to the next consumer
transform the data to a different type and push it to the next consumer
split the data into multiple chunks of data
drop the data and do not push it to the next consumer

A sink sits at the end of the pipeline and “drains” all events. It is the end of the process. After the sink processed the data the emitter emits the next event.

Consumer can provide various kind of functionality:

filter
aggregate
sum
group
distinct
map

Example

Emitter: Reads a file and pushes the data block by block to the next consumer Pipe: Calls the Qc3CalculateHash System API for every block of data Sink: Gets the calculated hash after the last block of data from the previous pipe and stores the hash

The code may look like this:

// setup the stream
stream = stream_create(emitter_file('data.txt')); // emits data from the file in blocks
stream_pipe(stream : pipe_md5());                 // calculates the md5 for the data block
stream_sink(stream : %paddr(writeMd5));           // local procedure storing the data

// start the stream , let the data flow
stream_flow(stream);

// cleanup the mess =)
stream_dispose(stream);

API Documentation

The API of this project is documented using ILEDocs and can be viewed at the ILEDocs installation at RPG Next Gen.

Installation

The project can either be build by using the make tool or installed with the iPKG client. It can be as simple as

ipkg install stream

The emitter will be packaged separately but can be as easily installed:

ipkg install 'stream-emitter'

No need to build from source.

Note: Use the LOC parameter on the ipkg command to specify where the object will be installed to.

The copybooks (prototypes, constants and templates) are packaged into separate packages.

stream-devel
stream-emitter-devel

Note: When installing the devel packages the LOC parameter accepts an IFS path for the copybook stream files.

Pipes will also be packaged in a separate package. But the project doesn’t contain any pipes yet.

Ideas

What can be further expected from the project? More pipes and emitters will definitely follow like data queue emitters or random number emitters.

The big goal is to be able to not just sequentially process the data but to be able to spawn new threads from a consumer and process the data in parallel.

Feedback

I am happy for every feedback and contribution.

Happy streaming!

Mihael

iPKG beta 2

Sat, 25 Apr 2020 00:00:00 +0200

Hi folks!

The next beta is out with some new features.

First a simple but important one: VERSION action. Now you can see which iPKG client version you got. Currently the version is stored in the user defined attribute of the object. But this can only store 10 characters and only one value. There is an RFE Add more user defined attributes/object information to an object out to extend that. Please vote for it so we can store more metadata directly on the object itself.

Back to business: Dependencies can now also be set to an installed product or PTF (thanks to Holger Scherer and Kevin Adler for their help). For more information see the wiki page Requirements. Example:

5770DG1_SI71714 = 7.4

There has also been more work on the opposite side where the package can define what it provides. By default a package provides itself. But now it is also possible to define more capabilities of a package on which a dependency can be expressed. F. e. a package needs a HTTP client as a dependency. It doesn’t matter which HTTP client as long as there is at least one installed. So it defines a requirement to httpclient. Welcome to the land of virtual packages.

And as I mentioned HTTP … you can now specify a repository with an HTTP or HTTPS protocol which means you don’t have to manually download and “install” the repository but can use the existing one on the web as is.

ipkg addrepo "RPGNextGen https://repo.rpngextgen.com/repository"

There has also been some bug fixes and some database changes so you need to remove the old iPKG tables and let the client install the new one.

I added the Java RPM library redline to the download section of the iPKG project. You can create packages programmatically with it. I also uploaded an example on how to do that.

I hope more people will give this project a try. ANY feedback is welcome.

BTW: Is there any interest into automatic building of RPM packages like this?

This could be coupled with the source code checkout and build of a project in a Jenkins pipeline which makes a wonderful CI/CD process =) .

Happy building and testing!

Mihael

iPKG ... package management for ILE in beta phase ... and now what?

Thu, 16 Apr 2020 00:00:00 +0200

Hi folks!

Happy Easter!

I am in the midst of developing an open source package management system for IBM i. Now many people may say: “We already have yum for managing packages on IBM i. I should get up-to-date.” Yes … and no.

Yes: yum manages packages
No: It does not manage packages for the whole IBM i system, but only those meant for PASE.

Where are the packages for ILE?

There is a lot of good open source software for ILE (which is a great environment by the way =) ). There was a discussion a decade ago about an app store for IBM i but that faded away as fast as it came and without any result at all (and with the demise of developerworks probably any traces have been erased).

So I set up a project and am now at a stage where the community can jump in and help. I got a client for managing packages. The client is a native ILE program (RPG) with a command object, see save file at the download section https://bitbucket.org/m1hael/ipkg/downloads/ . It is available for IBM i 7.3 (and of course compatible with 7.4).

I packaged some of my own projects. These packages are available at https://repo.rpgnextgen.com .

The client is at a late alpha/early beta stage I would say.

And this is the part where you can jump in: I need some testers and some feedback.

The project is hosted at https://bitbucket.org/m1hael/ipkg/ . To get into the project

Read the README.md
Read the wiki articles
Download and try out the client

Currently the client can only install files on the local file system. No installation vie http/https. But you can download the whole RPG Next Gen repository (just 1 MB) and put it in the IFS.

How to give feedback?

First things first: ANY feedback is welcome!

You can

write your feedback in the Ryver forum ( https://ibmioss.ryver.com/#posts/2360640 )
create a ticket at https://bitbucket.org/m1hael/ipkg/issues/
mail me directly : mihael@rpgnextgen.com

I don’t care which way you use. Just do it!

I will collect everything as tickets in Bitbucket because it will be easier to work with the feedback from there on.

Many thanks to those investing their precious time (which is one of the most precious resources you have). If you have any questions just ask!

Happy testing!

Mihael

Creating a Unix Timestamp

Wed, 15 Apr 2020 00:00:00 +0200

As most people know the Unix timestamp is an integer value which represents the seconds from 1970-01-01.

There is currently no built-in function for it but you can easily calculate it by yourself.

**FREE

ctl-opt dftactgrp(*no) actgrp(*caller);

dcl-pr sys_getUtcOffset extproc('CEEUTCO');
  offsetHours int(10);
  offsetMinutes int(10);
  offsetSeconds float(8);
  feedback char(12) options(*omit);
end-pr;
  
dcl-c UNIX_EPOCH_START z'1970-01-01-00.00.00.000000';

main();
*inlr = *on;


dcl-proc main;
  dcl-s uxts int(10);
  dcl-s now timestamp;
  dcl-s offsetHours int(10);
  dcl-s offsetMinutes int(10);
  dcl-s offsetSeconds float(8);
  
  now = z'2020-04-15-00.00.00.000';
  sys_getUtcOffset(offsetHours : offsetMinutes : offsetSeconds : *omit);
  
  uxts = %diff(now : UNIX_EPOCH_START : *SECONDS) - offsetSeconds;
  
  // Output : 1586908800
  dsply %char(uxts);
end-proc;

iPKG - Global = Local

Sat, 07 Mar 2020 00:00:00 +0100

In most contexts this is not true : Global = Local

But in the case of iPKG there is no difference between a global , for all users, installation of packages or a local installation (scoped to a single user) from the perspective of the package manager.

Objects get restored in a library and stream files in the IFS. It doesn’t matter to the package manager if that library is in some library list or not. So you can easily have one installation in a dedicated library for all users and have another installation in a different library with packages from different repositories (or the same) , f. e. for development purposes.

The client lets you decide where to put the packages, see parameter iPKG library or IPKGLIB. This is also the place where all the package management data for this installation will be stored (all objects starting with IPKG). But you can also split this and have the objects located to another library, see parameter location or LOC.

I know that some sites like to separate data objects from program objects and have their files, tables, views and indexes in a different library. This is also supported by iPKG.

Happy coding!

Mihael

Node.js Template Project

Sat, 15 Feb 2020 00:00:00 +0100

During my journey through the Node.js universe I have missed a good template for what I wanted to do. I searched for a simple template for a REST service on which I can build on.

You can start with this by following the next couple of steps and are ready to go

git clone https://m1hael@bitbucket.org/m1hael/template-rest.git
npm install
npm run build
npm start

This template also works out-of-the-box on IBM i.

I like the RESTED client for Firefox to test my services and also store the request for later reuse. The add on is also available for Chrome.

If you have started the service you can test it by making the following HTTP request:

GET http://localhost:3000/api/date?format=dd.MM.yyyy

The template also provides a Swagger interface for the REST service at http://localhost:3000/swagger .

The directory structure of the project is pretty straight:

config - configuration files
logs - logging directory (if not configured)
src - source code
- controller - REST endpoints
- docs - documentation
- middleware
- models - data models
- startup - scripts which are executed during service startup like configuring logging
- services

Application entry point is the src/server.ts file.

The template uses the config package for configuration. For most things a mapping to an environment variable is configured to keep it flexible.

The winston and morgan package has been used for logging. morgan has been used for creating an access log and winston for everything else. winston is configured to log in the logstash compatible format. The log entries also contains the application name as an additional field (see middleware/logapp.ts) so that you can easily filter for the application when using something like the ELK stack.

The Startup class also accepts an object in the constructor. This can be used to pass in or get out values to/from the startup scripts. Each startup script is an implementation of the Runnable interface.

During development I like to have at least two terminals open: one for compiling the application with “npm run build:watch” and another for running the application with “npm run watch” (which uses the nodemon package). This ensures a short code-compile-run-test development cycle.

Happy coding!

Mihael

libxlsxwriter - Date/Time and Image Support

Fri, 15 Mar 2019 00:00:00 +0100

libxlsxwriter is a great library for creating documents in the Open Document spreadsheet format.

Even so its code is very strict (and good) ANSI C code some things does not work out-of-the-box for integration in RPG and must be tweaked a little bit.

This was also the case for image and date/time support which has now been added to the forked repository at Bitbucket.org.

Image support is pretty simple. Currently only support for PNGs has been successfully tested and it looks like this:

worksheet_insert_image(worksheet : 3 : 2 : 'logo.png');

But you can also add some options like scaling or positional offsets.

dcl-ds imageOptions likeds(lxw_image_options) inz;
    
imageOptions.xScale = 0.5;
imageOptions.yScale = 0.5;
    
worksheet_insert_image_opt(worksheet : 3 : 2 : '/home/mihael/logo512.png' : imageOptions);

To support date/time in RPG we need an additional data structure which is not part of the original libxlsxwriter project, lxw_datetime.

dcl-ds datetime likeds(lxw_datetime) inz;
dcl-s dateFormat pointer;

dateFormat = workbook_add_format(workbook);
format_set_num_format(dateFormat : 'dd.mm.yyyy');

datetime = lxw_datetime_create(%date());
worksheet_write_datetime(worksheet : x : 2 : datetime : dateFormat);

And when it is finished it looks like this:

You can find the code for the above document here.

If you compile the project you should state your CCSID on the make command like

make -f Makefile.ibm TARGET_CCSID=273 env compile bind

For easier installation a save file and an archive file with the copybooks are available in the download section.

Happy spreadsheeting!

Mihael

Write Unicode Characters with libxlsxwriter

Wed, 27 Feb 2019 00:00:00 +0100

Hi,

the project libxlsxwriter lets your programmatically create spreadsheets in the Open Document format which is also used by OpenOffice, Microsoft Office, … .

Sometimes you need to add a special sign to some cell, f. e. the Euro currency symbol. If your native codepage does not support it you can still code it by escaping it. Characters can be escaped with the syntax _xHHHH_ where HHHH is the hex value of the unicode character, _x20AC_ for the euro sign.

This can again be escaped with escaping the first underscore with _x005F_. _x005F_x20AC_ will literally write the string and not the euro sign.

See Ecma Office Open XML Part 1 - Fundamentals and Markup Language Reference , 22.4.2.4 bstr (Basic String) at page 3736.

dcl-s moneyFormat pointer;

moneyFormat = workbook_add_format(workbook);
format_set_num_format(moneyFormat : '#,##0.00 "_x20AC_"');

worksheet_write_number(worksheet : 3 : 1 : 1499.99 : moneyFormat);

Happy escaping!

Mihael

Arraylist - Bugfix

Fri, 27 Jul 2018 00:00:00 +0200

A little memory leak has crept into the code.

The arraylist_dispose() method did not deallocate all the memory which has been previously allocated. This has been fixed now in the current version.

Every user of the JSON service program also needs to update the arraylist service program as it is used as a backend for the JSON array implementation.

You can find the new version on the project page at Bitbucket.

Happy updating!

Mihael

BlueDroplet - Next Steps

Thu, 21 Jun 2018 00:00:00 +0200

For those of you who don’t know yet: BlueDroplet is a native REST framework for all ILE languages on IBM i. BlueDroplet consists of multiple ILE modules which are bound to an ILE program. Here is an overview of the structure of such a program.

BlueDroplet is a work-in-progress. Some rudimentary things already work but other required things for a “complete” REST microservice still doesn’t work, f. e. PUT and POST requests.

Though if the service is only about providing data and not taking data as input you are ready to go with BlueDroplet.

Is BlueDroplet production-ready? No. But it is an already useful framework none-the-less. You can take it to get your feet wet and your hands dirty with developing your first REST microservice on IBM i without spending a dime. You also don’t have to get some trial license and don’t have to get and install a licensed program. Just download the source and compile it.

With BlueDroplet you can learn what it takes to build such a service. And what is most important is to realize what works for you and your environment and what doesn’t. F. e. how “big” should a model be? Or do you need more specific/smaller data models which may provide you with more flexibility. If the model is too small the client may need to make many requests to the server before having anything useful to display to the user.

Smaller models may also mean more microservices. This requires more attention in the administration.

You also learn how to organize your code so that you can use it for the REST service and also use it for your 5250 programs. Same code base. Just wrap it up in service programs.

There are many articles and blogs about REST services. Take a broader view and explore the internet for information about REST. A good source of information for me has been DZone.

And why now such an article about BlueDropet? Yes, for some time now the framework was able to successfully process GET requests. But now it also supports processing the URL path and query parameters. This is essential in communicating with modern web frameworks because many use JSONP. To provide the data as JSONP the REST service needs to inspect the query parameters and use the callback parameter in the request response.

F. e. the URL of a HTTP request with such a callback parameter may look like this:

http://localhost:8484/customer/123?callback=angular.callbacks._0

You can get the callback query parameter value with this code:

callbackId = droplet_http_getQueryValue(httpMessage : 'callback');

The REST service now builds the return data and wraps the callback parameter value around it like this:

angular.callbacks._0({'id':123, 'name':'Michael Meier'})

You can also query the path elements like this:

customerId = %int(droplet_http_getPathElement(httpMessage : 1));

So what are the next steps?

As we have now support for getting the path elements and query parameter values I can pick up development of the show case for BlueDroplet again, see project Store Management.

So get your hands dirty and take some REST. ;-)

Mihael

Linked List Release 1.4.0

Sun, 10 Jun 2018 00:00:00 +0200

The Linked List service program has been a very stable service program with a stable API.

The only thing that was a little bit clumsy was the sort function. The API design of the function offered great flexibility. You could implement your own sorting with your own sort implementation. The base sort algorithm used is the insertion sort. But the API offered the possiblity to use any other sort algorithm (if implemented).

That was all nice and good. But the more simple use case got more complicated:

Sorting the list in a user defined way (using the base sort algorithm).

For example if you add some data structure to the list and you want to sort the list on a subfield of the data structure. In the previous version you would copy the whole existing sort procedure and change the part where the values are compared. I consider this copy & paste development style a really bad style and it leads to many errors.

The new API makes it a lot easier to sort the list entries by just writing a compare procedure.

The prototype for such a procedure looks like this:

dcl-pr compare int(10) extproc(*dclcase);
  value1 pointer const;
  length1 int(10) const;
  value2 pointer const;
  length2 int(10) const;
end-pr;

The result of the compare procedure is inspired by the way most implementations handle this (C, Java, …).

0 means the values are equal (and the order of the list probably won’t change)
less than 0 means that the first value of the compare function should be higher in the list than the second value
greater than 0 means that the second value should be higher in the list than the first value

This makes comparing numbers fairly easy. For example the comparision of two integers.

dcl-proc compareIntegers export;
  dcl-pi *n int(10);
    value1 pointer const;
    length1 int(10) const;
    value2 pointer const;
    length2 int(10) const;
  end-pi;
  
  dcl-s x1 int(10) based(value1);
  dcl-s x2 int(10) based(value2);
  
  return x1 - x2;
end-proc;

But in the current implementations the pointer to the values can also be *null. So we must handle *null values in our compare procedure which bloats it a little bit.

dcl-proc compareIntegers export;
  dcl-pi *n int(10);
    value1 pointer const;
    length1 int(10) const;
    value2 pointer const;
    length2 int(10) const;
  end-pi;
  
  dcl-s x1 int(10) based(value1);
  dcl-s x2 int(10) based(value2);
  
  if (value1 = *null and value2 = *null);
    return 0;
  elseif (value1 = *null);
    return 1;
  elseif (value2 = *null);
    return -1;
  else;
    return x1 - x2;
  endif;  
end-proc;

And if you would like to have a descending order than just flip the expression to

return x2 - x1;

Another change is the way escape messages are sent (or more precisely where they are sent to). Up until now they were sent to the boundary of the current program. This is not always very practicle and has now been changed to the be sent to the current call stack entry.

The API change can be viewed in the new release and the documentation has been updated on the ILEDocs website.

Happy sorting!

Mihael

ILEDocs Database is growing again!

Tue, 29 May 2018 00:00:00 +0200

ILEDocs has some fresh additions, namely the API documentation of the projects JSON Utilities, Linked List and Linked Map.

We have almost all of the projects on board which are currently on the old documentation site at Sourceforge.net.

If you have an open source library and would like to add it to the ILEDocs documentation site just drop me a mail and I will help you with it.

Happy documenting!

Mihael

JDBCR4 and Batch Updates and Inserts

Wed, 04 Apr 2018 00:00:00 +0200

I had to work with a remote Oracle database lately and the choice was made to use Scott's wrapper around the Java JDBC API.

The solution worked. Everything is fine. Is it? No it isn't.

I had to insert some thousand records into the Oracle database. What JDBCR4 supports is something like

statement = JDBC_PrepStmt(connection : 'INSERT INTO ...');

for i = 1 to numberRecords;
  JDBC_setString(statement : 1 : data.name);
  JDBC_ExecPrepUpd(statement);
endfor;

The data is processed record by record. That would be the same as using embedded SQL to fetch a set of data but fetching the data row by row instead of using a multi row fetch to get a whole block of records with one call.

In Java you can do the same by batching SQL statements. You need some more prototypes for that.

dcl-c JDBC_STATEMENT_SUCCESS_NO_INFO -2;

dcl-pr PreparedStatement_addBatch extproc(*java : 'java.sql.PreparedStatement' : 'addBatch') end-pr;
dcl-pr PreparedStatement_executeBatch int(10) dim(999)
           extproc(*java : 'java.sql.PreparedStatement' : 'executeBatch') end-pr;

Now the workflow looks like this:

statement = JDBC_PrepStmt(connection : 'INSERT INTO ...');

for i = 1 to numberRecords;
  JDBC_setString(statement : 1 : data.name);
  PreparedStatement_addBatch(statement);
endfor;

resultData = PreparedStatement_executeBatch(statement);

If this really speeds up execution depends on the JDBC driver (and probably the database).

Happy batching!

Mihael

ILEDocs Database is growing!

Mon, 26 Mar 2018 00:00:00 +0200

Slowly but steadily the ILEDocs project is growing. Today with the new version of the MESSAGE service program the API documentation has been uploaded to the ILEDocs database, making the API publicly available without having to search through any source codes.

What does that mean for the ILE developers? You can find the MESSAGE API at the central place for documentation about open source libraries for IBM i.

If you have an open source library and would like to add it to the ILEDocs documentation site just drop me a mail and I will help you with it.

Happy documenting!

Mihael

libxlsxwriter - Creating spreadsheets natively

Wed, 21 Mar 2018 00:00:00 +0100

A short while ago I stumbled over the C library libxlsxwriter from John McNamara on github. It is a C library for creating XLSX documents. More specifically it creates files in the Open Document spreadsheet format. Which can be used by Microsoft Excel but also by OpenOffice and every application which supports the Open Document standard.

Technically the file is a ZIP archive which contains various XML files and may contain some embedded resources like images.

This library makes it so easy to create such a document.

This whole project would not have interested me so much if it wasn’t for the very strict ANSI C standard which has been used and the very clean API and overall design and implementation. It also has very minimal requirements and dependencies.

The only third party dependency which is not shipped with the library is zlib which can be compiled on the platform very nicely. zlib even had an upgrade to a nicer compile script. Thanks for making things easier.

The other dependency (minizip) can also be compiled to an ILE C module.

And thanks to the great coding the whole libxlsxwriter project can be compiled to ILE C modules and bound to a service program. This means that now every ILE programming language is able to create spreadsheets. =)

The character encoding was a little problematic because everything is written with the default locale which results in most cases in files with an EBCDIC CCSID. I have extend the code of the library to convert the data to UTF-8.

Also the formatting of floating point values was a little problematic as Excel expects the values to have a point as a decimal point. But many locales have a comma as a decimal point sign. To overcome this problem the environment variable LC_NUMERIC can be set to an English locale.

ADDENVVAR ENVVAR(LC_NUMERIC) VALUE('/QSYS.LIB/EN_US.LOCALE')

Note: The DECFMT job value does not change the decimal point sign in this case as the function snprintf() does not take the job value DECFMT into account but the environment variable LC_NUMERIC.

The RPG prototypes for C functions can be found in the include directory of the project.

Creating a spreadsheet looks like this:

/include 'xlsxwriter_h.rpgle'

dcl-proc main;
  dcl-s workbook pointer;
  dcl-s worksheet pointer;

  workbook = workbook_new('/home/schmidtm/test1.xlsx');

  worksheet = workbook_add_worksheet(workbook : 'Testsheet 1');
 
  worksheet_write_number(worksheet : 0 : 0 : 358 : *null);
  worksheet_write_number(worksheet : 1 : 0 : 35,8 : *null);
  worksheet_write_string(worksheet : 0 : 1 : 'Hello World!' : *null);
  worksheet_write_string(worksheet : 1 : 1 : 'Umlaute: äöüÄÖÜ and ß' : *null);

  workbook_close(workbook);
end-proc;

The library can be installed by entering QShell (qsh) and using the Makefile.ibm script like this

make -f Makefile.ibm

This compiles all necessary modules and binds them to a service program. You should adjust the values at the top of the Makefile.ibm to your environment.

On systems configured with EBCDIC 37 this should work out-of-the-box. On other systems you may need to convert the source file packager.c to your CCSID as there are characters in the code which will be used in filenames (specifically brackets [ and ] ) and these may get trashed in the compile step.

The library has many features. I plan to add the prototypes step by step to the project. Currently I added only the rudimentary features to the RPG prototypes.

Images are currently also not supported in my fork of the project (but are supported in the original code on github).

The project is currently hosted on Bitbucket.

Happy spreadsheeting!

Mihael

RDi Hover Support

Wed, 07 Mar 2018 00:00:00 +0100

At the moment I have access to an RDi installation and use it on a daily basis (besides using MiWorkplace =) ).

Some people have written about the new hover support for RDi which is quite nice (and MiWorkplace has also a nice hover support to some extent) but I haven’t read or heard about a very nice feature of it:

Procedure documentation!

Now you can put your procedure documentation before your prototype and have it displayed in the hover info window.

Example code:

///
// Write Number
//
// Writes a number into the specified cell in the worksheet.
//
// @param worksheet
// @param row
// @param column
// @param number
// @param format
///
dcl-pr worksheet_write_number int(10) extproc('worksheet_write_number');
  worksheet pointer value;
  row int(10) value;
  column int(10) value;
  number float(8) value;
  format pointer value;
end-pr;

The hover info for this prototoype looks like this:

The information is not formatted but perhaps that will be implemented in the future (*hint* RFE ;-) ). It would also be nice if it would support the ILEDocs tags but as IBM has no interest in such things support for such will probably never come.

Perhaps this feature will add a little more to the poor motivation of writing documentation. Because now you have a benefit of the documentation immediately right at your fingertips without wading through some office documents. Just hover with the mouse over the procedure name.

Happy hovering and documenting!

Mihael

Measuring Time Spent

Mon, 12 Feb 2018 00:00:00 +0100

Sometimes you have a block of code and just want to know how long it took the machine to execute it.

The Java developers have some really cool metric libraries like the Dropwizard subproject Metrics.

But for the quick and dirty case there is always

System.currentTimeMillis();

For the RPG developer there are some built-in functions which can be used for this, but nothing is as straightforward as the Java solution.

But if we tap into the powers of ILE and use some C functions in RPG we can come very close to the Java solution.

The C function time() returns the Unix timestamp in an integer. The prototype looks like this.

1
2
3

dcl-pr time int(10) extproc('time');
  returnValue pointer;
end-pr;

The function also takes a parameter for output but we don’t need this and can pass *NULL for it.

dcl-s NULL pointer;
dcl-s start int(10);

start = time(NULL);

dsply %trimr('Time spent: ' + %char(time(NULL) - start));

Note: The resolution of time() is seconds. If milliseconds are needed another function should be used.

Happy measuring!

Mihael

Project CodeNotes

Sat, 27 Jan 2018 00:00:00 +0100

At the time I still developed the RPG Next Gen Editor I have received some requests regarding offline support. It was not as much about editing the code but more about being able to view the code (perhaps for some reviewing and later refactoring).

About 10 years forward I am picking up the topic again. There still isn’t much support for offline viewing (though there some nice web tools for reviewing code online). But much has changed in the IT world with everyone and everything going mobile. So I am going mobile with this project too.

The goal of the project is an Android app with full code review capabilities. Including

getting the code from various sources
viewing the code
taking notes in context to whole code / section of the code
share notes
create tickets from notes in Bitbucket, GitHub, GitLab, Taiga.io, Trac, Redmine …

The whole project is not just an idea but already has some code behind it. There is a beta version to download from the website. It already has some functionality. You can download code from a Git repository (I have tested Bitbucket and SourceForge ). Files from a file archive (like a zip or tar archive file) can also be fetched. But I really want to have IBM i support as a first class citizen in this project. You can browse the QSYS filesystem and the IFS and download code to your Android device.

And the price for it …. nothing. Just download it from the website. If you want to show me your appreciation then send me a mail and/or make a donation on rpgnextgen.com (top right corner on the web site).

For everybody who needs another feature which is not covered by the current state of the project:

Good news! You can even contribute to the project!

It is open source after all. Take the source work on it and don’t forget to contribute back to the project.

So … what do you think about it?

Happy reviewing!

Mihael

Modular Programming

Sat, 28 Oct 2017 00:00:00 +0200

Modular programming is a software design technique that emphasizes separating the functionality of a program into independent, interchangeable modules, such that each contains everything necessary to execute only one aspect of the desired functionality. — from Wikipedia

I think that is a good explanation and a design technique which should be used (probably) always and every time though on different scales.

Globals

Each time we use global variables it probably goes against this software design technique and we try to minimize the use of global variables. I say “minimize” because in RPG it is not always possible, f. e. data areas have to be defined globally.

And everybody seem to agree to this concept on this scale because now we don’t have to scan all 20000 lines of code if a certain variable is used but only the 100 lines of the procedure. Everyone can see the benefit of it. Impact analysis is made in seconds (or minutes) instead of hours and we don’t need any expensive tools for doing it.

Modular at System Level

But for some reason most RPG developers find it ok that a program should access any table regardless of the domain (order, master data, …). It should read and write data from/to it no matter what failing to see the resemblence to global variables. The table is the global variable in the system everyone has access to from everywhere. So now when we need to make a change to some table what is the scope of the impact analysis we have to do?

THE WHOLE SYSTEM!

Could we have minimized the scope of it: Definitely!

How? Offer access to the data via some API. But by offering some API I don’t mean to just add a database layer (serviceprogram/procedure) between your program and the table/data without adding any value to it. Just by wrapping your table access into another layer you have gained nothing. Any change behind the scenes of the API should not automatically lead to a change of the API consumer. If it does then you should think about what you have gained by using the API at all.

Now any change to the table should only require an analysis of the program code of the corresponding domain and not the whole system.

In some cases the scope of the analysis can even be minimized to a single service program which makes it even easier.

What are you talking about?

Many programmers don’t realize this because they have written the system themselves or lived in the system long enough to know it. But for everybody new to the system it is really hard to make even the tiniest change. Those systems are really hard and costly to maintain because everybody need to know the whole system.

The database should NOT be your API!

Your situation?

Just ask yourself what do you need to know/analyse if you want to make a change? One procedure? One program? Code of one domain? The whole system?

Processing XML Data

Tue, 10 Oct 2017 00:00:00 +0200

For some time now I had the idea to port a REST service of mine from Java to RPG. Mostly just out of curiosity because the REST service was running well enough. The REST service was about serving the geo coordinates of the boundaries of postalcode areas.

Import

The data source for these coordinates is the web site suche-postleitzahl.org. They generously supply us with the latest geographic data of Germany. Thanks guys. Keep it going =) .

The data is in the KML format which is an XML based format. So we can process the file with native RPG tools, in our case the opcode XML-SAX. Boundaries are made up of multiple coordinates. The coordinates for one boundary are located in the KML file as one long string of characters. The string can have a length of over 75000 characters. And lately (probably since IBM i 6.1) we can declare variables big enough to hold the content in RPG. But what about storing such a big character string in the database. The CHAR data type has a max. size of 32k characters (plus some more but that doesn’t matter here). But like so many other databases the DB 2 for i also supports the data type CLOB (Character Large Object). So this should work out of the box with embedded SQL.

But like so often … it won’t work out of the box. CHAR and CLOB are two fundamentally different data types and are not compatible in embedded SQL. The following code looks nice and straight forward but doesn’t work.

1
2
3

dcl-s coords char(100000);
EXEC SQL INSERT INTO postalcodes(country, postalcode, city, coords)
         VALUES('DE', :postalcode, :city, :coords);

Since V5R4 RPG supports working with LOBs with the RPG data type SQLTYPE. At first this didn’t seem to work out in this case at all because most tutorials and information about RPG and CLOBs were written when it came out with V5R4. But at that time RPG still had the limit for CHAR fields of 65k and this was also pointed out in the articles. So this wouldn’t help because the coordinate data is bigger than 65k. But that was at V5R4. Since then much has happened and now you can declare CHAR variables with up to 16m characters.

1
2
3

dcl-s coords sqltype(clob : 100000);
EXEC SQL INSERT INTO postalcodes(country, postalcode, city, coords)
         VALUES('DE', :postalcode, :city, :coords);

REST Service

The REST service part of it would be implemented by using the ILE REST framework BlueDroplet. But some features are not implemented yet in the framework and so this part has to wait a little bit longer.

Source Code

You can take a look at the code here.

Notes

Saving the coordinates in the format from the KML file is not ideal in any way because now every time a boundary is requested the string of coordinates need to be converted into the desired format. It would be much better to convert them directly on the import so we would have to convert them once instead of x times. But that has to wait till the BlueDroplet project has proceeded further and has implemented the feature to parse the query string of an HTTP request. So that we can really use this data and return it not as JSON but as JSONP which makes it much easier to access and process it in the fronted (JavaScript).

Mihael

BlueDroplet at ILEDocs

Mon, 02 Oct 2017 00:00:00 +0200

As there is now a functioning ILEDocs site available at rpgnextgen.com I added some real content to it.

The BlueDroplet project is now in an alpha stage and can already be used for some REST services. The ILEDocs documentation of the BlueDroplet gives you a nice and quick overview of the available API from BlueDroplet and helps you developing your REST services with BlueDroplet.

What do you think about the documentation. Anything missing? I would like to hear your thoughts and feedback.

Happy browsing!

Mihael

Even more Integration Possibilities

Sat, 26 Aug 2017 00:00:00 +0200

IBM i stands for integration. But it is most of the time very troublesome to connect to and integrate anything outside the IBM family. In the open world (outside the IBM i box) there are numerous ways to integrate and communicate between systems and applications. One way to do it is to use a messaging system. There are numerous messaging systems out there (like ActiveMQ, Artemis, Apollo from Apache or RabbitMQ to name only a few). Even IBM has such a system by the name of IBM MQ (short names must be the way-to-go at the moment as it was formerly named IBM Websphere MQ and MQSeries before that. IBM has a long track record of renaming things and we all know , don’t we?! ;-) ).

But for RPG developers there was no easy way to communicate with these systems. Most programming language have some libraries which can be used to communicate with these systems in an easy way. RPG as so often doesn’t have one.

Most of these messaging systems support the STOMP protocol. That is a simple text protocol which can be used to talk to these systems.

From the former STOMP web site (at codehaus.org):

The STOMP project is the Streaming Text Orientated Messaging Protocol site (or the Protocol Briefly Known as TTMP and Represented by the symbol :ttmp). STOMP provides an interoperable wire format so that any of the available Stomp Clients can communicate with any STOMP Message Broker to provide easy and widespread messaging interop among languages, platforms and brokers.

So putting together a client library for RPG should not be that big of a problem and after all it wasn’t (except for some stumbling blocks when it came to socket APIs).

The client library is available as an ILE service program written in RPG at its project website hosted on Bitbucket.

Sending Data

A common format should be chosen for the data which is sent to the messaging system. Currently JSON and XML are very common and easy to process formats.

Basically the following steps are needed to send data to a messaging system:

Create an instance of the STOMP client:

1	client = stomp_create('mq.server.com' : 61613);

Connect on the network level (socket connection):

1	stomp_open(client);

Connect on the application level (STOMP protocol):

1	stomp_command_connect(client : 'user' : 'pass');

Send the data:

1	stomp_command_send(client : '/queue/items' : '{ "id":5500 , "price":1.29 }');

Disconnect on the application level (STOMP protocol):

1	stomp_command_disconnect(client);

Disconnect on the network client (close socket connection):

1	stomp_close(client);

Clean up and free all allocated memory:

1	stomp_finalize(client);

At the moment all data is sent as ASCII. The STOMP protocol (at least in its recent version) also supports other encodings but that is not implemented yet (contributions are welcome =) ).

Documentation

The API documentation has been created with ILEDocs and can (soon) be viewed here.

Examples

The project repository has an examples folder which contains some code on how to use the procedures.

There is also a complete demo project available on Bitbucket which shows how to use the procedures.

Help

If you need any help with setting things up or on how to use the service program feel free to contact me at mihael@rpgnextgen.com .

Mihael

Hey Socket, take a break!

Thu, 24 Aug 2017 00:00:00 +0200

Ever wanted a normal (blocking) socket connection to listen for data but also wanted to check for a condition if it should stop listening?

Problem is that the default BSD 4.3 version of the socket API (which is the default API to bind to) does not support setting timeouts. Trying to set a timeout for receiving data results in

3021 : The value specified for the argument is not correct.

But by switching to the BSD 4.4 / Unix98) version of the socket API you can set timeouts with the setsockopt function.

How to do it

The timeout can be set directly after the connect() call. It is set by calling the setsockopt() function with the corresponding parameters.

Parameters:

Socket descriptor
SOL_SOCKET for stating that it will be a socket option
SO_RCVTIMEO for changing the timeout for receiving data
timeout value
timeout value length

The prototype looks Like this:

dcl-pr setsockopt int(10) extproc('qso_setsockopt98');
  socket int(10) value;
  level int(10) value;
  option_name int(10) value;
  option_value pointer value;
  option_length int(10) value;
end-pr;

The timeout is passed as a data structure containing seconds and microseconds for the timeout value.

dcl-ds timeout_t qualified template;
  seconds int(10);
  useconds int(10);
end-ds;

The call may look something like this:

if (setsockopt(socket : SOL_SOCKET: SO_RCVTIMEO : %addr(timeout) : %size(timeout)) = -1);
  errPtr = errno();
  message_sendEscapeMessageToCaller(
        'Could not set socket timeout. ' + %char(c_err) + ': ' + %str(strerror(c_err)));
endif;

Binding

The socket functions are contained in the service programs QSOSRV1. The contained modules and functions can be displayed with DSPSRVPGM QSOSRV1.

The BSD 4.3 and 4.4 versions of the Socket API are contained in this service program. The BSD 4.4 version (which is also identical to the Unix98 version) is prefixed with qso_ and suffixed with 98.

So the program can be created in the usual way and be bound to the same service program. The names in the prototypes are important as they decide which functions are called.

Example

For a working example take a look at the STOMP client, source file stomp.rpgle, procedure stomp_open.

Have a nice break.

Mihael

Rant about XMLSERVICE

Sun, 21 May 2017 00:00:00 +0200

Hi,

First of all I want to make it clear that I have nothing against XMLSERVICE. I think it is a nice tool though I never had to work with it due to my current job and thus have no real life experience with it.

And this ranting is not really about the product itself but more about the marketing and what the marketing does to the IBM i community.

So what bugs me most is the marketing of “Now you can offer REST services to the world!”. If it would have been marketed not as REST services but as web services there wouldn’t be anything to rant about because there is no real/strict definition of what a web service really is (though many people have some strong expectations when they hear about web services).

The W3C defines a web service generally as:

A web service is a software system designed to support interoperable machine-to-machine interaction over a network.

And I want to throw in that I am no REST service expert by any means but even myself with my intermediate REST service skills I see many flaws in the way it is marketed.

First of all REST is all about the resource and the resource is stated in the URL. So if I want some information about a book the URL may look like this:

http://ibmi/book/1234

where 1234 is the id of the book.

With XMLSERVICE you call the global XMLSERVICE program. You don’t call the resource you want to know about.

http://ibmi:58700/cgi-bin/xmlcgi.pgm

Another thing is that the HTTP method is directly linked to the action you want to execute on the resource. GET retrieves some information and DELETE deletes the information on the server. With XMLSERVICE there is no such mapping to HTTP methods. AFAIK there are only GET and POST supported.

What about the media type? Normally you would want to support at least JSON and XML and perhaps even some other format. But the media type does not only state the expected return format of the information. It can contain much more, f. e. the API version.

application/vnd.company.book-v3+xml

HTTP Status Codes … what about the returned status codes. I would expect a 404 if the book is not in the database of the server. What about authentifiation and authorization status codes? Or if the user cannot update the resource because it is blocked by some other transaction? I would want the user to be informed about that. Just throwing some 500 or not even at all (actually I don’t know what XMLSERVICE does in these cases) is not good. Fact is that the program which handles the resoure should set the HTTP status code and not XMLSERVICE.

There are probably many other details that are not correct when calling XMLSERVICE a REST service provider. But that is not really the point here. In calling XMLSERVICE a REST provider or a framework/toolkit which can provide REST services (which it obviously doesn’t) you make it look non-professional. For someone really deep into REST services this more or less looks like a joke (which it definitely isn’t). But developers from other platform are laughing about this:

A: “Hey, you know … IBM i now can provide their data as REST services. *giggle*“

B: “Yeah … *giggle* … at least I can get the data with some HTTP call.”

So calling XMLSERVICE a framework for creating REST services is also adding to how other developers (from other platforms) see the IBM i platform (and i don’t mean in a positive way).

The only technical argument I have with XMLSERVICE: Actually I don’t understand the rational about the design concept of XMLSERVICE. In (almost) every language there is the trend to make small libraries which do exactly one thing and with as little dependencies as possible to build modular applications. But XMLSERVICE is the exact opposite.

If you can’t be mainstream stop pretending to be mainstream when you obviously are not.

And again I would like to mention that I find it great to have XMLSERVICE because it adds to the small list of tools and libraries available to IBM developers whereas developers on other platforms can’t even count the number of tools/libraries available to them.

End of my rant.

Mihael

Continuous Integration on IBM i

Sun, 14 May 2017 00:00:00 +0200

Hi,

continuous integration has been a topic on any other platform for some years now. But on IBM i I never heard that term in any discussion.

There is a myriad of open source tooling support on any platform but not for IBM i … or rather there is tooling for anything but for ILE on IBM i.

The central piece of software for CI is a build server which runs the build jobs but can also execute other tasks like unit and integration test and can also be used to automatically deploy the build to a test or integration environment. Jenkins is such a build server.

I have written an article about the results of my adventures with Jenkins and some RPG code. You can find it in my wiki.

Happy integrating!

Mihael

Arraylist vs Linked List

Sun, 19 Feb 2017 00:00:00 +0100

Hi,

I am just moving my service program Linked List to the OSSILE project and I am again wondering if people know why there are two list implementations and when to use which implemenation or when to use a list at all.

I am doing most of my development in the Java world and there lists are really cheap. You will encounter them in Java everywhere. In the RPG world it is exactly the other way around. Arrays are really cheap and there is not even a list data
structure natively in RPG.

Note: And by data structure I don't mean a DS or dcl-ds declared variable but something more general as explained on this page.

So using a list is not as cheap as using an array from a performance point of view. But it definitely has its pros. If you don't know the size of the result you want to return from a procedure you are at a loss when using an array because no matter how hard you try you cannot do it correctly. The number of returned elements will almost never match the size of your array. The array will either be too big or may be even too small (which would be rather fatal).

And this was my main motiviation of implementing a list. Because with a list the number of elements doesn't matter. The list starts with the number of elements you declare (probably none) and grows as you add elements to it. Depending on the available API it may also have some very convenient procedures like rotate all elements in the list, remove a range of elements, split a string of characters by one separator to a list, last index of element, swap elements, … .

Arraylist vs Linked List

The main difference of these lists is the way the data is stored and accessed.

Arraylist

The Arraylist service program uses one big block of data to store the pointers to each element. By doing that it can really quickly access elements by index. Adding elements to the end of the list is performing normally but adding to the beginning of the list is a bad idea performance wise because all pointers to the other elements must be shifted down by one position. Also adding elements could result in the need to expand this block of memory where the pointers are stored. This operation may have a bigger impact on performance. Iterating through the nodes is really easy as it can jump to the next node by just one arthimetic expression.

Linked List

The Linked List service program is implemented as a two-way linked list where each node has a pointer to the previous node and to the next node. Because of this inserting a node at any place takes the same time no matter how large the list grows. Iterating through the list is also very cheap as you go from node to node. But accessing the list by index is not a good idea because the list has to start at the beginning of the list and then count till it reaches the node at the specified index.

Not good at

Both lists are not good at doing one thing: locating an element by value. So if you want to check if a value is already in the list then you can do it. Both lists support it by the contains procedure but both lists have to start from the beginning of the list and check every element if it has the same value as the passed one. For this kind of operation these two lists are the wrong data structure or rather the wrong implementation to use. You would need something like a sorted list where all entries are sorted at any time. Or use a tree data structure.

Wrap it up

If you know the size of the content you should use an array.
If you don't know the size of an array use a list.
If you need to access the list by index use the Arraylist.
If you have many insert and remove operations on the list use the Linked List.

I hope that gave you a good idea about when to use a list and which implementation to use.

Happy coding!

Mihael

BlueDroplet and the library list

Fri, 10 Feb 2017 00:00:00 +0100

BlueDroplet now supports changing the library list on startup. You can either add library list entries to the user portion of the library list or replace the whole user portion of the library list.

Happy RESTing!

Mihael

ILEDocs Revival

Thu, 09 Feb 2017 00:00:00 +0100

Hi,

creating documentation has always been a major PITA in software development. Tools try to ease the process of creating good documentation. There are different kind of documentations (mostly different by the scope and/or detail and/or perspective) and not every documentation tool is good for every kind of documentation.

API documentation is one kind of documentation. It is rather easy to create in most programming languages as there are many good toolchains like Javadoc or doxygen. Sadly for development on IBM i you will not find a good toolchain or documentation support on our beloved platform for your own development (I am not talking about the IBM i Information Center).

I tried to tackle this problem once with a not so pretty solution. Especially installation and maintenance was not very well thought out. Editor support was also not very good or even non-existent.

Now some years later and with new technologies appearing I have a feeling that I can tackle this problem much better and easier than before. I revived the ILEDocs project with a fresh approach using Dropwizard for the backend and AngularJS for the frontend. A MongoDB database will store the documentation.

Updating the documenation is now tooling independant as we can work with a REST service and just need to upload a JSON document.

The MiWorkplace editor will get some extra features like querying the documentation database and updating the documentation directly from the editor.

The documentation could also be updated from some workflow like a TurnOver form run utilizing the old ILEDocs RPG and CL parser, converting the parse result to JSON and sending it with the HTTP Client to the ILEDocs REST service. Though this is not implemented.

The project is not finished but already quite usable. It is open source and available under the Apache License 2.0.

You can take a look at it at bitbucket.org. The wiki at the bitbucket project site contains some useful information on how to setup your own documentation server.

One of my goal is also to create a universal documentation server for all open source IBM i projects. Please contact me if you would like to help.

A prototype is also already online with some demo content at iledocs.rpgnextgen.com. Feel free to give it a try.

I would like to hear your thoughts and feedback.

Happy documenting!

Mihael