Geek Answers Handbook

How to debug apache storm in local cluster / mode via eclipse

Using the following Q & A, I was able to get debugging through eclipse in an Apache Storm cluster (running locally). How to debug apache storm in eclipse?

My conf/storm.yaml has the following line to enable debugging on work nodes:

 worker.childopts: "-agentlib:jdwp=transport=dt_socket,address=8000,server=y,suspend=y"

When I send the Topology to storm for launch (in the cluster), I can set breakpoints and view the variables in my editor.

But when I try to run it locally (in Local mode ), I can not connect (Connection Refused) - through an eclipse.

 # I'm using storm crawler, I submit a topology like so: storm jar target/storm-crawler-core-10.6-SNAPSHOT-jar-with-dependencies.jar \ com.digitalpebble.storm.crawler.CrawlTopology \ -conf crawler-conf.yaml \ -local # ^ The `-local` runs it in a `LocalCluster` # If I submit it to my actual cluster (without -local), I can debug it through eclipse. # View the pastebin for all the output : http://pastebin.com/PEdA7fH0

I have included all the output from the above command in pastebin. Click here to view it.

Additional information on running the LocalCluster trap.

I want to be able to debug in local mode so that I can see the output on the command line (when I go through the breakpoints that I set), and quickly make changes and restart, mainly to speed up my development flow.

How can I debug my code through the eclipse debugger when running Apache Storm in LocalCluster (local mode) ?

+1
source share
3 answers

If you work in local mode, the JVM worker is not involved, that is, the workflow is not visible. Thus, your worker.childopts settings have no effect.

The easiest way to debug in Eclipse is to send / run the topology in Eclipse instead of the command line. Your CrawlTopology class has a main method, so you can simply execute it directly in Eclipse (of course, in debug mode). You do not need to specify a jar file name. Just specify your -conf crawler-conf.yaml -local in the Eclipse launch configuration.

+3
source

We can debug the Storm topology like any other Java code in Eclipse. I run my topology in Eclipse itself using LocalCluster mode. I can debug ShellBolts, Trident bolts, etc.

In addition, I also have a storm kernel added as a project in my Eclipse (instead of a maven dependency), which allows me to debug and understand what is happening inside the storm. It is useful to improve tracking of Tuple confirmations, to issue handlers, bindings, timeouts, exceptions, etc.

0
source

In local mode, you can use, for example,

export STORM_JAR_JVM_OPTS = "- agentlib: jdwp = transport = dt_socket, server = y, suspend = y, address = localhost: 8000"

before invoking the jar storm command to debug remotely using Eclipse.

0
source

More articles:


All Articles