Issue
I have 1 web and 1 worker process communicating with each other using CloudAMQP
library consisting of 2 channels/queues (1 channel is to send a simple message from web
, and the other channel is to receive a message from worker
). Since I'm deploying my webapp in WAR
file format, I had to put my worker
class into a jar by creating a local repository of Maven and including dependency to put the file in WEB-INF/lib
folder so that I could execute the worker process using the generated worker sh
script from maven-war-plugin
artifact.
worker sh
script :
BASEDIR=`dirname $0`/..
BASEDIR=`(cd "$BASEDIR"; pwd)`
# OS specific support. $var _must_ be set to either true or false.
cygwin=false;
darwin=false;
case "`uname`" in
CYGWIN*) cygwin=true ;;
Darwin*) darwin=true
if [ -z "$JAVA_VERSION" ] ; then
JAVA_VERSION="CurrentJDK"
else
echo "Using Java version: $JAVA_VERSION"
fi
if [ -z "$JAVA_HOME" ] ; then
JAVA_HOME=/System/Library/Frameworks/JavaVM.framework/Versions/${JAVA_VERSION}/Home
fi
;;
esac
if [ -z "$JAVA_HOME" ] ; then
if [ -r /etc/gentoo-release ] ; then
JAVA_HOME=`java-config --jre-home`
fi
fi
# For Cygwin, ensure paths are in UNIX format before anything is touched
if $cygwin ; then
[ -n "$JAVA_HOME" ] && JAVA_HOME=`cygpath --unix "$JAVA_HOME"`
[ -n "$CLASSPATH" ] && CLASSPATH=`cygpath --path --unix "$CLASSPATH"`
fi
# If a specific java binary isn't specified search for the standard 'java' binary
if [ -z "$JAVACMD" ] ; then
if [ -n "$JAVA_HOME" ] ; then
if [ -x "$JAVA_HOME/jre/sh/java" ] ; then
# IBM's JDK on AIX uses strange locations for the executables
JAVACMD="$JAVA_HOME/jre/sh/java"
else
JAVACMD="$JAVA_HOME/bin/java"
fi
else
JAVACMD=`which java`
fi
fi
if [ ! -x "$JAVACMD" ] ; then
echo "Error: JAVA_HOME is not defined correctly."
echo " We cannot execute $JAVACMD"
exit 1
fi
if [ -z "$REPO" ]
then
REPO="$BASEDIR"/repo
fi
CLASSPATH=$CLASSPATH_PREFIX:"$BASEDIR"/etc:"$REPO"/com/rabbitmq/amqp-client/3.3.4/amqp-client-3.3.4.jar:"$REPO"/javax/servlet/jstl/1.2/jstl-1.2.jar:"$REPO"/com/example/worker/1.0/worker-1.0.jar:"$REPO"/cloudamqp/example/amqpexample/1.0-SNAPSHOT/amqpexample-1.0-SNAPSHOT.war
EXTRA_JVM_ARGUMENTS=""
# For Cygwin, switch paths to Windows format before running java
if $cygwin; then
[ -n "$CLASSPATH" ] && CLASSPATH=`cygpath --path --windows "$CLASSPATH"`
[ -n "$JAVA_HOME" ] && JAVA_HOME=`cygpath --path --windows "$JAVA_HOME"`
[ -n "$HOME" ] && HOME=`cygpath --path --windows "$HOME"`
[ -n "$BASEDIR" ] && BASEDIR=`cygpath --path --windows "$BASEDIR"`
[ -n "$REPO" ] && REPO=`cygpath --path --windows "$REPO"`
fi
exec "$JAVACMD" $JAVA_OPTS \
$EXTRA_JVM_ARGUMENTS \
-classpath "$CLASSPATH" \
-Dapp.name="worker" \
-Dapp.pid="$$" \
-Dapp.repo="$REPO" \
-Dbasedir="$BASEDIR" \
WorkerProcess \
"$@"
The worker sh
script is then referenced into Heroku Procfile
:
web: java $JAVA_OPTS -jar target/dependency/webapp-runner.jar --port $PORT target/*.war
worker: sh target/bin/worker
I tested the app locally using heroku local -f Procfile.windows
, and all works as expected. However, when I deploy to Heroku (I have tried using both Heroku CLI deployment and git
as well), only the web
Procfile can be executed properly. The worker process gives back error 127
when I check the log, saying it cannot open target\bin\worker
script. I've tried to google what error 127
for heroku means, but so far there's no match, and I have no idea what causes of my remotely deployed apps not being able to open target\bin\worker
even though it works fine in my local deployment. Any help will be appreciated.
Below here are the full error logs from Heroku, servlet class, worker class, and POM respectively :
2016-10-31T14:27:32.629768+00:00 heroku[slug-compiler]: Slug compilation finished
2016-10-31T14:27:34.807925+00:00 heroku[web.1]: State changed from down to starting
2016-10-31T14:27:34.941663+00:00 heroku[worker.1]: State changed from down to starting
2016-10-31T14:27:37.334213+00:00 heroku[worker.1]: Starting process with command `sh target/bin/worker`
2016-10-31T14:27:37.825597+00:00 heroku[worker.1]: State changed from starting to up
2016-10-31T14:27:38.100208+00:00 app[worker.1]: sh: 0: Can't open target/bin/worker
2016-10-31T14:27:38.104979+00:00 heroku[worker.1]: State changed from up to crashed
2016-10-31T14:27:38.104979+00:00 heroku[worker.1]: State changed from crashed to starting
2016-10-31T14:27:38.291551+00:00 heroku[web.1]: Starting process with command `java $JAVA_OPTS -jar target/dependency/webapp-runner.jar $WEBAPP_RUNNER_OPTS --port 54481 target/amqpexample-1.0-SNAPSHOT.war`
2016-10-31T14:27:40.725610+00:00 app[web.1]: Setting JAVA_TOOL_OPTIONS defaults based on dyno size. Custom settings will override them.
2016-10-31T14:27:40.732741+00:00 app[web.1]: Picked up JAVA_TOOL_OPTIONS: -Xmx350m -Xss512k -Dfile.encoding=UTF-8
2016-10-31T14:27:41.385588+00:00 app[web.1]: Expanding amqpexample-1.0-SNAPSHOT.war into /app/target/tomcat.54481/webapps/expanded
2016-10-31T14:27:41.385722+00:00 app[web.1]: Adding Context for /app/target/tomcat.54481/webapps/expanded
2016-10-31T14:27:41.817336+00:00 heroku[worker.1]: Starting process with command `sh target/bin/worker`
2016-10-31T14:27:42.172560+00:00 app[web.1]: INFO: Initializing ProtocolHandler ["http-nio-54481"]
2016-10-31T14:27:42.172549+00:00 app[web.1]: Oct 31, 2016 2:27:42 PM org.apache.coyote.AbstractProtocol init
2016-10-31T14:27:42.201060+00:00 app[web.1]: Oct 31, 2016 2:27:42 PM org.apache.tomcat.util.net.NioSelectorPool getSharedSelector
2016-10-31T14:27:42.201064+00:00 app[web.1]: INFO: Using a shared selector for servlet write/read
2016-10-31T14:27:42.205036+00:00 app[web.1]: Oct 31, 2016 2:27:42 PM org.apache.catalina.core.StandardService startInternal
2016-10-31T14:27:42.205038+00:00 app[web.1]: INFO: Starting service Tomcat
2016-10-31T14:27:42.206587+00:00 app[web.1]: Oct 31, 2016 2:27:42 PM org.apache.catalina.core.StandardEngine startInternal
2016-10-31T14:27:42.206588+00:00 app[web.1]: INFO: Starting Servlet Engine: Apache Tomcat/8.0.30
2016-10-31T14:27:42.378327+00:00 heroku[worker.1]: State changed from starting to up
2016-10-31T14:27:42.441771+00:00 app[web.1]: Oct 31, 2016 2:27:42 PM org.apache.catalina.startup.ContextConfig getDefaultWebXmlFragment
2016-10-31T14:27:42.441781+00:00 app[web.1]: INFO: No global web.xml found
2016-10-31T14:27:42.765321+00:00 heroku[web.1]: State changed from starting to up
2016-10-31T14:27:43.507545+00:00 app[worker.1]: sh: 0: Can't open target/bin/worker
2016-10-31T14:27:43.591348+00:00 heroku[worker.1]: Process exited with status 127
2016-10-31T14:27:43.603552+00:00 heroku[worker.1]: State changed from up to crashed
2016-10-31T14:27:44.410603+00:00 app[web.1]: Oct 31, 2016 2:27:44 PM org.apache.jasper.servlet.TldScanner scanJars
2016-10-31T14:27:44.410621+00:00 app[web.1]: INFO: At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
2016-10-31T14:27:44.498715+00:00 app[web.1]: Oct 31, 2016 2:27:44 PM org.apache.coyote.AbstractProtocol start
2016-10-31T14:27:44.498718+00:00 app[web.1]: INFO: Starting ProtocolHandler ["http-nio-54481"]
2016-10-31T14:27:47.410949+00:00 heroku[router]: at=info method=GET path="/" host=amqpexample.herokuapp.com request_id=334f7e05-d4e5-4f78-a444-1e6e7403e52c fwd="14.192.210.168" dyno=web.1 connect=1ms service=4305ms status=200 bytes=370
2016-10-31T14:27:50.003897+00:00 heroku[router]: at=info method=GET path="/" host=amqpexample.herokuapp.com request_id=51c9fac7-6763-4374-9f2b-672f0d7f7be7 fwd="14.192.210.168" dyno=web.1 connect=1ms service=21ms status=200 bytes=295
Servlet class (responsible to a display a buffer message and waiting for new message to be sent from worker
when being refreshed) :
package herokutest;
import com.rabbitmq.client.*;
import java.io.IOException;
import java.net.URISyntaxException;
import java.security.KeyManagementException;
import java.security.NoSuchAlgorithmException;
import javax.servlet.ServletConfig;
import javax.servlet.ServletException;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
public class HelloWorld extends javax.servlet.http.HttpServlet {
private final static String QUEUE_NAME = "hello";
private final static String ANOTHER_QUEUE = "ANOTHER";
public Channel channel;
public Channel anotherChannel;
public DefaultConsumer consumer;
public String message = "Temporary";
public void init(final ServletConfig config) throws ServletException {
try {
createChannel();
} catch (Exception e) {
e.printStackTrace();
}
finally {
consumer = new DefaultConsumer(anotherChannel) {
@Override
public void handleDelivery(String consumerTag, Envelope envelope, AMQP.BasicProperties properties, byte[] body) throws IOException {
String innerMessage = new String(body, "UTF-8");
message = innerMessage;
}
};
try {
anotherChannel.basicConsume(ANOTHER_QUEUE, true, consumer);
} catch (IOException e) {
e.printStackTrace();
}
}
}
public void createChannel() throws NoSuchAlgorithmException, KeyManagementException, URISyntaxException, IOException {
String uri = System.getenv("CLOUDAMQP_URL");
if (uri == null) uri = "amqp://guest:guest@localhost";
ConnectionFactory factory = new ConnectionFactory();
factory.setUri(uri);
factory.setRequestedHeartbeat(30);
factory.setConnectionTimeout(30);
Connection connection = factory.newConnection();
channel = connection.createChannel();
anotherChannel = connection.createChannel();
channel.queueDeclare(QUEUE_NAME, false, false, false, null);
anotherChannel.queueDeclare(ANOTHER_QUEUE, false, false, false, null);
}
protected void doGet(final HttpServletRequest request, final HttpServletResponse response) throws ServletException, IOException {
String test="test";
channel.basicPublish("", QUEUE_NAME, null, test.getBytes());
request.setAttribute("message", message);
request.getRequestDispatcher("/index.jsp").forward(request, response);
}
}
Worker class : (responsible for sending a message Another message
back to servlet)
import com.rabbitmq.client.*;
import java.io.IOException;
public class WorkerProcess {
private final static String QUEUE_NAME = "hello";
private final static String ANOTHER_QUEUE = "ANOTHER";
static String message;
public static void main(String[] argv) throws Exception {
ConnectionFactory factory = new ConnectionFactory();
String uri = System.getenv("CLOUDAMQP_URL");
if (uri == null) uri = "amqp://guest:guest@localhost";
factory.setUri(uri);
factory.setRequestedHeartbeat(30);
factory.setConnectionTimeout(30);
Connection connection = factory.newConnection();
Channel channel = connection.createChannel();
final Channel anotherChannel = connection.createChannel();
channel.queueDeclare(QUEUE_NAME, false, false, false, null);
anotherChannel.queueDeclare(ANOTHER_QUEUE, false, false, false, null);
System.out.println(" [*] Connected to Worker");
final Consumer consumer = new DefaultConsumer(channel) {
@Override
public void handleDelivery(String consumerTag, Envelope envelope, AMQP.BasicProperties properties, byte[] body) throws IOException {
message = new String(body, "UTF-8");
String anotherMessage= "Another message";
anotherChannel.basicPublish("", ANOTHER_QUEUE, null, anotherMessage.getBytes());
}
};
channel.basicConsume(QUEUE_NAME, true, consumer);
}
}
POM :
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>cloudamqp.example</groupId>
<artifactId>amqpexample</artifactId>
<version>1.0-SNAPSHOT</version>
<packaging>war</packaging>
<dependencies>
<dependency>
<groupId>com.rabbitmq</groupId>
<artifactId>amqp-client</artifactId>
<version>3.3.4</version>
</dependency>
<dependency>
<groupId>javax.servlet</groupId>
<artifactId>jstl</artifactId>
<version>1.2</version>
</dependency>
<dependency>
<groupId>javax.servlet</groupId>
<artifactId>javax.servlet-api</artifactId>
<version>3.0.1</version>
<scope>provided</scope>
</dependency>
<!-- This is my worker jar included as dependency -->
<dependency>
<groupId>com.example</groupId>
<artifactId>worker</artifactId>
<version>1.0</version>
</dependency>
</dependencies>
<!-- My local repository containing my worker .jar file-->
<repositories>
<repository>
<id>project.local</id>
<name>project</name>
<url>file:${project.basedir}/localrepo</url>
</repository>
</repositories>
<build>
<plugins>
<plugin>
<groupId>com.heroku.sdk</groupId>
<artifactId>heroku-maven-plugin</artifactId>
<version>1.1.1</version>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-war-plugin</artifactId>
<version>2.4</version>
<configuration>
<webResources>
<resource>
<directory>src/web</directory>
</resource>
</webResources>
<archiveClasses>false</archiveClasses>
</configuration>
</plugin>
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>appassembler-maven-plugin</artifactId>
<version>1.1.1</version>
<configuration>
<assembleDirectory>target</assembleDirectory>
<programs>
<program>
<mainClass>WorkerProcess</mainClass>
<name>worker</name>
</program>
</programs>
</configuration>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>assemble</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-dependency-plugin</artifactId>
<version>2.3</version>
<executions>
<execution>
<phase>package</phase>
<goals><goal>copy</goal></goals>
<configuration>
<artifactItems>
<artifactItem>
<groupId>com.github.jsimone</groupId>
<artifactId>webapp-runner</artifactId>
<version>8.0.30.2</version>
<destFileName>webapp-runner.jar</destFileName>
</artifactItem>
</artifactItems>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
</project>
Solution
This is the important message in the logs:
2016-10-31T14:27:43.507545+00:00 app[worker.1]: sh: 0: Can't open target/bin/worker
Does your maven build generate this target/bin/worker
script?
I would not expect the CLI deploy to work because it does not include the target
dir by default. But you can include it with the --include
option.
Answered By - codefinger