[FIXED] Java jar execution stuck when launched in a process by another java program

Issue

This is a very unusual problem I've come across and I'm hoping someone might have some insight on it. I'm on macOS Mojave (10.14.6), using Amazon's JRE Corretto-11.0.9.12.1 (build 11.0.9.1+12-LTS)

I have a program I've written that is something of a scripting engine that we use to do our routine bulk data processing. It takes an xml "script" file which contains the processing directions to execute and a list of input files as arguments. I'll refer to this as the "engine" from this point on. This engine is the backbone of a large portion of our automation. It shows a progress bar while processing to let users know that it is working.

There are 2 programs that use this engine:

• One is a thin UI written in Swing, which we use to manually process data; it generates an xml file from the user input and passes it along with the input files and launches the engine in a separate process; the UI itself doesn't process any data.
• The other watches a folder on our file server and processes incoming data from our clients daily when a folder is created inside of it so we can rip the data into our database. I'll call this the "importer".

Recently, a problem has come up where the engine becomes stuck while processing. Older versions of the engine did not have this problem, and I'm trying to figure out what exactly changed that caused this to happen, but while I've been trying to do this, our UI and importer programs have been using and older version of the engine. There are new features that we need to use in the new version of the engine, but we can't use it until this problem is solved.

The programs that uses the engine launch it in a process then waits for the result before continuing:

// example command generated from external program
String commandString = "java -jar engine.jar script.xml input_file1.txt input_file2.txt input_file3.txt";
String[] command = {"bash", "-c", commandString};
// I can grab the command from here for debugging
System.out.println(Arrays.toString(command));
ProcessBuilder pb = new ProcessBuilder(command);
// wait for the process to complete before continuing
Process p = pb.start();
p.waitFor();
int result = p.exitValue();
try (BufferedReader e = new BufferedReader(new InputStreamReader(proc.getErrorStream()));
    BufferedReader i = new BufferedReader(new InputStreamReader(proc.getInputStream()))) {
    String line;
    while ((line = e.readLine()) != null) {
        System.out.println(line);
    }
    while ((line = i.readLine()) != null) {
        System.out.println(line);
    }
}
p.destroy();
// do other stuff

When launched in this way, there is a specific operation that causes the engine to hang. But if I take the command and launch it directly from the command line, the engine runs just fine! This is making it difficult to pin down where exactly the problem is; is it in the engine, or in the other programs? I've spent a couple of days looking for answers and come up with nothing. It's even more frustrating that this problem has appeared seemingly out of nowhere when it was working perfectly before, using the exact code above, for a quite a long time.

The operation where the engine hangs sorts files into folders based on their file names. When I watch my activity monitor while it runs, it's not taxing my resources at all, and disk space isn't an issue. It isn't a file permission issue, as the engine is creating files and folders all the time and in every step leading up to the step where it hangs. And as I said, if I run the command directly from the command line, it creates the folders and sorts the files without issue, to my extreme confusion.

The importer and UI run locally on a station, but the engine jar file lives on our file server, so that it is accessible to every station without individually downloading it everywhere each time there is an update. I thought at first that the issue might lie in the fact that it is being accessed over the network, but the problem occurs even when I use a local copy of the engine on my dev machine, so I have ruled that out. I've also ruled out that it's the JRE, even though we switched to it recently, since the older version of the engine still perform as expected.

Solution

There might of course be any reason why your 'engine' program may hang ;-) but certainly it will hang you don't read the its output, and in the right way:

The parent process needs to read the standard output and standard error streams of the child process, given that the child process does generate any substantial amount of output on any of these two channels. This must be done in two separate background threads. If the parent does not read the child's output, then the child process will block as soon as the (small) buffer between the processes is filled up.

The threads should be started as soon as the child process is started, and before the parent calls process.waitFor().

The simplest way to do this is the following:

    Process process = processBuilder.start();
    InputStream stdout = process.getInputStream();
    InputStream stderr = process.getErrorStream();

    Thread stdoutThread = new Thread() {
        @Override
        public void run() {
            // read stdout here, e.g.
            try {
                int c;
                while (-1 != (c = stdout.read())) {
                    // do whatever with c
                }
            } catch (IOException ioex) {
                // ...
            }
        }
    };

    Thread stderrThread = new Thread() {
        // ... same as for stdout
    };

    stdoutThread.start();
    stderrThread.start();
}

Only after both threads have been started you may wait for the child process and join the threads:

    int exitValue = process.waitFor();
    stdoutThread.join();
    stderrThread.join();

There might be more sophisticated ways to work with background threads using the Concurrency Framework introduced in Java 5, but this basic code gives the idea.

Answered By - gsl
Answer Checked By - Terry (JavaFixing Volunteer)