Replacement for obsolete `WrappingNeoServerBootstrapper`

Question

I am trying to convert Java Neo4J code code to 2.2+ format, the code starts a built-in Neo4J server with a web interface.

// configure embedded DB, // but this doesn't start a server at port 12345 ? final GraphDatabaseService db = new GraphDatabaseFactory() .newEmbeddedDatabaseBuilder("/path/to/db") .setConfig(ServerSettings.webserver_address, "localhost") .setConfig(ServerSettings.webserver_port, 12345) .newGraphDatabase(); // add shutdown hook Runtime.getRuntime().addShutdownHook(new Thread(){ @Override public void run() { graphDb.shutdown(); } }); 

The outdated code that starts the server reads:

 final GraphDatabaseAPI api = (GraphDatabaseAPI)db; final ServerConfigurator c = new ServerConfigurator(api); c.configuration().addProperty( Configurator.WEBSERVER_ADDRESS_PROPERTY_KEY, "localhost"); c.configuration().addProperty( Configurator.WEBSERVER_PORT_PROPERTY_KEY, 12345); new WrappingNeoServerBootstrapper(api, c).start(); 

Leaving this, the server does not start. I don’t understand how to start the embedded server without using obsolete methods. Any thoughts?

UPDATE

So, the really correct answer is: change your architecture.

You cannot use the embedded server, use the server and the RESTful API instead. For bits in which you need the Java API, you can write an unmanaged extension (see docs ).

Here is a simple example of such an extension https://github.com/histograph/neo4j-plugin

Thanks!

Answer 1

UPDATE January 2016

In the comments, you will notice that some things about neo4j are changing. The rest of this answer, in my opinion, is still valid, but if your traveler-traveler, who came to this post in mid-2016, examines whether the new binary protocol neo4j (AFAIK, called the "bolt") is available.

/ EndUpdate

The bad news is that the neo server boostrapper wrap is deprecated, and I don't think there is a way to do this without using obsolete methods. Therefore, the simple answer to your question is no.

Perhaps a year ago I was looking for the same thing as you, but let me share what I learned, in the meantime, that could change your question.

Therefore, when you use the built-in server, you get certain advantages, such as the ability to use the java API, but you get some big disadvantages. Putting a database in the same memory space as your application means that you have to juggle memory between them. And this means that you cannot update the application without stopping the database.

When you use the server, many things are better than the ability to run cypher without going through your application level, using RESTful services, etc.

The problem with the WrappingNeoServerBootstrapper (and why it is probably good that it is out of date) is that it has the disadvantages of both approaches. Of course, you get REST and cypher services, but you are still in the same memory space as the application.

Architecturally, what we found with neo4j is that you usually want to use an external separate server and then exchange data using REST services. I think neo4j is working on a binary protocol (ala JDBC, but certainly not the same), so maybe the java API will expand more and performance will improve. In the meantime, all the good libraries for neo4j, including spring-data and others, can already talk to the HTTP endpoint, so any functions they provide can usually be executed using a separate server. To ensure the functionality of the application and configure neo4j itself, you will almost always be better off running a separate server.

Please note that this recommendation mimics what you see in almost all other types of databases (mongo, oracle, whatever). Some offer built-in options for small / specialized applications, but almost all of them assume that you start a separate server process and ask the application to talk to it. Thus, we are not talking about graphics in general, but about good application design, evolution and maintainability.

So, for these reasons, it is normal that the WrappingNeoServerBootstrapper is deprecated (and probably not coming back). My experience shows that this is not what you really need to do anyway.

Answer 2

You can always do it the other way around. Instead of passing the instance of GraphDatabaseService that you created on the server, first try to start the server and get the instance created by it :) Tested with 3.0.6

  String homeDir = "./"; String configFile = "./conf/neo4j.conf"; ServerBootstrapper serverBootstrapper = new CommunityBootstrapper(); int i = serverBootstrapper.start(new File(homeDir), Optional.of(new File(configFile)), Pair.of("dbms.connector.http.address","0.0.0.0:7575")); NeoServer neoServer = serverBootstrapper.getServer(); GraphDatabaseService graph = neoServer.getDatabase().getGraph(); System.out.println(graph); 

Answer 3

Czajah, Hi, your workaround for 3.0.6 also works in native mode (older paxos HA). 3.1.0 - thanks.

A source

The github project (Maven, Tomcat 8 WAR) at https://github.com/obrienlabs/nbi-neo4j-embedded-aws-war

Endpoints

Add node to inline db graph http://neo4j.ca-central-1.elasticbeanstalk.com/FrontController?action=graph

The browser is on port 7575

With a few changes, I was able to wrap the HighlyAvailableGraphDatabase database with EnterpriseBootstrapper.

There are some non-fatal exceptions around JMX reports in debug.log, which are probably related to my version of tomcat 8.0.28, but the db graph built into Tomcat is fine.

  2016-12-21 16:20:00.574+0000 INFO Bolt enabled on 0.0.0.0:7688. 2016-12-21 16:20:09.554+0000 INFO Attempting to join cluster of [127.0.0.1:5001] 2016-12-21 16:20:11.566+0000 INFO Creating new cluster with name [neo4j.ha]... 2016-12-21 16:20:11.607+0000 INFO Instance 1 (this server) entered the cluster 2016-12-21 16:20:12.164+0000 INFO I am 1, moving to master 2016-12-21 16:20:12.293+0000 INFO Instance 1 (this server) was elected as coordinator 2016-12-21 16:20:12.462+0000 INFO I am 1, successfully moved to master 2016-12-21 16:20:12.513+0000 INFO Instance 1 (this server) is available as master at ha://127.0.0.1:6001?serverId=1 with StoreId{creationTime=1482199697648, randomId=7800059877674392627, storeVersion=15531981201765894, upgradeTime=1482199697648, upgradeId=1} 2016-12-21 16:20:14.495+0000 INFO Database available for write transactions 2016-12-21 16:20:31.917+0000 INFO Mounted REST API at: /db/manage 2016-12-21 16:20:53.264+0000 INFO Remote interface available at http://localhost:7575/ register: org.obrienlabs.nbi.graph.service.HaMonitor@1c0f80c9 public class ExtendedHighlyAvailableGraphDatabaseFactory extends HighlyAvailableGraphDatabaseFactory { private Log log = LogFactory.getLog(ExtendedHighlyAvailableGraphDatabaseFactory.class); private HaMonitor haMonitor; @Override public GraphDatabaseService newDatabase( final Map<String, String> config ) { EnterpriseBootstrapper serverBootstrapper = new EnterpriseBootstrapper(); List<Pair> pairs = new ArrayList<>(); for(Entry<String, String> entry : config.entrySet()) { pairs.add(Pair.of(entry.getKey(), entry.getValue())); } Pair pairArray[] = new Pair[pairs.size()]; // will resolve to /dir/data/databases/graph.db serverBootstrapper.start(storeDir, Optional.empty(), pairs.toArray(pairArray)); GraphDatabaseService graph = serverBootstrapper.getServer().getDatabase().getGraph(); // set the paxos HA listener only when dbms.mode=HA if(graph instanceof HighlyAvailableGraphDatabase) { haMonitor.setDb((HighlyAvailableGraphDatabase) graph); HighAvailabilityMemberStateMachine memberStateMachine = ((HighlyAvailableGraphDatabase)graph).getDependencyResolver() .resolveDependency(HighAvailabilityMemberStateMachine.class); if ( memberStateMachine != null ) { memberStateMachine.addHighAvailabilityMemberListener(haMonitor); log.info("register: " + haMonitor); } } return graph; } }; } 

Spring config

  <util:map id="config"> <entry key="ha.server_id" value="1"/> <entry key="ha.initial_hosts" value="127.0.0.1:5001"/> <entry key="dbms.mode" value="HA"/> <entry key="browser.allow_outgoing_connection" value="true" /> <entry key="unsupported.dbms.ephemerall" value="false" /> <entry key="dbms.connector.http.address" value="0.0.0.0:7575" /> <entry key="dbms.connector.bolt.address" value="0.0.0.0:7688" /> <entry key="dbms.connector.http.enabled" value="true" /> <entry key="dbms.connector.bolt.enabled" value="true" /> <entry key="dbms.connector.http.type" value="HTTP" /> <entry key="dbms.connector.bolt.type" value="BOLT" /> <entry key="dbms.connector.http.tls_level" value="DISABLED" /> <entry key="dbms.connector.bolt.tls_level" value="DISABLED" /> <entry key="dbms.security.auth_enabled" value="true"/> <entry key="dbms.logs.debug.level" value="DEBUG"/> <entry key="dbms.logs.http.enabled" value="true" /> <entry key="dbms.logs.query.enabled" value="true"/> <entry key="dbms.shell.enabled" value="true"/> </util:map> <bean id="haMonitor" class="org.obrienlabs.nbi.graph.service.HaMonitor"/> <bean id="graphDbFactory" class="org.obrienlabs.nbi.graph.service.ExtendedHighlyAvailableGraphDatabaseFactory"> <constructor-arg ref="haMonitor" /> </bean> <bean id="graphDbBuilder" factory-bean="graphDbFactory" factory-method="newEmbeddedDatabaseBuilder"> <constructor-arg value="/ec2-user"/> </bean> <bean id="graphDbBuilderFinal" factory-bean="graphDbBuilder" factory-method="setConfig"> <constructor-arg ref="config"/> </bean> <!-- HighlyAvailableGraphDatabase wrapped by an EnterpriseBootstrapper NeoServer created in this constructor --> <bean id="graphDatabaseService" factory-bean="graphDbBuilderFinal" factory-method="newGraphDatabase" destroy-method="shutdown" /> 

Transactions go through

 2016-12-21 20:51:07.478+0000 INFO [onkiscCountsTracker] About to rotate counts store at transaction 9 to [/ec2-user/data/databases/graph.db/neostore.counts.db.b], from [/ec2-user/data/databases/graph.db/neostore.counts.db.a]. 2016-12-21 20:51:07.480+0000 INFO [onkiscCountsTracker] Successfully rotated counts store at transaction 9 to [/ec2-user/data/databases/graph.db/neostore.counts.db.b], from [/ec2-user/data/databases/graph.db/neostore.counts.db.a]. 2016-12-21 20:51:07.483+0000 INFO [onkitlcCheckPointerImpl] Check Pointing triggered by scheduler for time threshold [9]: Store flush completed 

Settings from mbean to jconsole

 2016-12-23 13:28:12.708+0000 INFO [onkiDiagnosticsManager] Kernel version: 3.1.0,16a782b42d76ca37db72958eb2565cf6aa671a29 2016-12-23 13:28:12.708+0000 INFO [onkiDiagnosticsManager] Neo4j Kernel properties: 2016-12-23 13:28:12.708+0000 INFO [onkiDiagnosticsManager] dbms.logs.query.enabled=true 2016-12-23 13:28:12.708+0000 INFO [onkiDiagnosticsManager] dbms.security.auth_enabled=true 2016-12-23 13:28:12.708+0000 INFO [onkiDiagnosticsManager] dbms.connector.bolt.tls_level=DISABLED 2016-12-23 13:28:12.708+0000 INFO [onkiDiagnosticsManager] dbms.shell.enabled=true 2016-12-23 13:28:12.708+0000 INFO [onkiDiagnosticsManager] ha.server_id=1 2016-12-23 13:28:12.708+0000 INFO [onkiDiagnosticsManager] dbms.logs.http.enabled=true 2016-12-23 13:28:12.708+0000 INFO [onkiDiagnosticsManager] ha.initial_hosts=127.0.0.1:5001 2016-12-23 13:28:12.708+0000 INFO [onkiDiagnosticsManager] dbms.allow_format_migration=false 2016-12-23 13:28:12.708+0000 INFO [onkiDiagnosticsManager] dbms.connector.http.address=0.0.0.0:7575 2016-12-23 13:28:12.708+0000 INFO [onkiDiagnosticsManager] dbms.connector.bolt.enabled=true 2016-12-23 13:28:12.708+0000 INFO [onkiDiagnosticsManager] dbms.connector.http.tls_level=DISABLED 2016-12-23 13:28:12.708+0000 INFO [onkiDiagnosticsManager] dbms.connector.http.enabled=true 2016-12-23 13:28:12.708+0000 INFO [onkiDiagnosticsManager] dbms.mode=HA 2016-12-23 13:28:12.708+0000 INFO [onkiDiagnosticsManager] unsupported.dbms.block_size.labels=56 2016-12-23 13:28:12.708+0000 INFO [onkiDiagnosticsManager] dbms.connector.bolt.type=BOLT 2016-12-23 13:28:12.708+0000 INFO [onkiDiagnosticsManager] unsupported.dbms.directories.neo4j_home=/ec2-user 2016-12-23 13:28:12.708+0000 INFO [onkiDiagnosticsManager] unsupported.dbms.ephemerall=false 2016-12-23 13:28:12.708+0000 INFO [onkiDiagnosticsManager] browser.allow_outgoing_connection=true 2016-12-23 13:28:12.708+0000 INFO [onkiDiagnosticsManager] dbms.logs.debug.level=DEBUG 2016-12-23 13:28:12.708+0000 INFO [onkiDiagnosticsManager] dbms.connector.http.type=HTTP 2016-12-23 13:28:12.708+0000 INFO [onkiDiagnosticsManager] unsupported.dbms.block_size.strings=120 2016-12-23 13:28:12.708+0000 INFO [onkiDiagnosticsManager] unsupported.dbms.block_size.array_properties=120 2016-12-23 13:28:12.708+0000 INFO [onkiDiagnosticsManager] unsupported.dbms.edition=enterprise 2016-12-23 13:28:12.708+0000 INFO [onkiDiagnosticsManager] dbms.connector.bolt.address=0.0.0.0:7688 

7575 (berth port) and ports 7688 (bolt) are open

 obrienlabs-mbp15:_deployment michaelobrien$ netstat -vatn | grep 7575 tcp46 0 0 *.7575 *.* LISTEN 131072 131072 49013 0 tcp4 0 0 127.0.0.1.7575 127.0.0.1.60685 TIME_WAIT 407296 146988 49013 0 tcp6 0 0 ::1.7575 ::1.60699 TIME_WAIT 407284 146808 49013 0 tcp6 0 0 ::1.7575 ::1.60700 TIME_WAIT 407284 146808 49013 0 obrienlabs-mbp15:_deployment michaelobrien$ netstat -vatn | grep 7688 tcp6 0 0 ::1.7688 ::1.60704 ESTABLISHED 406582 146808 49013 0 tcp6 0 0 ::1.60704 ::1.7688 ESTABLISHED 398196 146808 48165 0 tcp6 0 0 ::1.7688 ::1.60702 ESTABLISHED 406570 146808 49013 0 tcp6 0 0 ::1.60702 ::1.7688 ESTABLISHED 398185 146808 48165 0 tcp6 0 0 ::1.7688 ::1.60701 ESTABLISHED 407255 146808 49013 0 tcp6 0 0 ::1.60701 ::1.7688 ESTABLISHED 407628 146808 48165 0 tcp46 0 0 *.7688 *.* LISTEN 131072 131072 49013 0 obrienlabs-mbp15:_deployment michaelobrien$ netstat -vatn | grep 8080 tcp4 0 0 127.0.0.1.8080 127.0.0.1.60584 FIN_WAIT_2 408104 146988 49013 0 tcp4 994 0 127.0.0.1.60584 127.0.0.1.8080 CLOSE_WAIT 408128 146988 42992 0 tcp46 0 0 *.8080 *.* LISTEN 131072 131072 49013 0 udp4 0 0 *.* *.* 196724 9216 38080 0 

/ec2-user/logs/http.log

 2016-12-23 02:53:21.505+0000 INFO [REQUEST] [AsyncLog @ 2016-12-23 02:53:21.505+0000] 0:0:0:0:0:0:0:1 - [Thu Dec 22 21:53:21 EST 2016] "/browser/views/frame-cypher.html?null" 200 22972 "http://localhost:7575/browser/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_1) AppleWebKit/602.2.14 (KHTML, like Gecko) Version/10.0.1 Safari/602.2.14" 2 

query.log

 2016-12-23 13:33:39.059+0000 INFO 175 ms: bolt-session bolt neo4j neo4j-javascript/[object Object] client/0:0:0:0:0:0:0:1:64019 server/0:0:0:0:0:0:0:1:7688> neo4j - MATCH (a)-[r]->(b) WHERE id(a) IN {node_ids} AND id(b) IN {new_node_ids} RETURN r; - {node_ids: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22], new_node_ids: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22]} - {} 

Disclaimer: I believe that the correct architecture includes a cluster set of servers with support for Neo4j server mode using causal clustering using the raft protocol. This undocumented / internal API is used as a temporary workaround to get the neo4j browser on the integrated Neo4j server in version 3.1.0, since it was used to work in WrappingNeoServerBootstrapper in version 2.3. Using an embedded server for faster crawls was a precedent - performance should be reevaluated to see if bolt-based architecture is more appropriate for the crawl API. /Michael

Answer 4

With the Neo4J 3.1.1 and czajah approaches, the "built-in" mode can really be launched just fine (without outdated calls), as well as with the advantage of manipulating the graph data from the same JVM instance.

The sample code below requires artifacts of the dependencies neo4j and neo4j-server on org.neo4j . The server configuration file is omitted, and the working storage folder and the corresponding configuration parameters are specified so that the neo4j console and neo4j-shell (which is becoming obsolete in favor of cypher-shell ) can be used against the "embedded" instance.

 import org.neo4j.graphdb.GraphDatabaseService; import org.neo4j.helpers.collection.Pair; import org.neo4j.server.CommunityBootstrapper; import org.neo4j.server.NeoServer; import org.neo4j.server.ServerBootstrapper; import java.io.File; import java.io.IOException; import java.util.Optional; public class RunNeoRun { public static void main(String[] args) throws IOException { // Wherever the Neo4J storage location is. File storeDir = new File("/tmp/tmp4j"); ServerBootstrapper serverBootstrapper = new CommunityBootstrapper(); serverBootstrapper.start( storeDir, Optional.empty(), // omit configfile, properties follow Pair.of("dbms.connector.http.address","127.0.0.1:7474"), Pair.of("dbms.connector.http.enabled", "true"), Pair.of("dbms.connector.bolt.enabled", "true"), // allow the shell connections via port 1337 (default) Pair.of("dbms.shell.enabled", "true"), Pair.of("dbms.shell.host", "127.0.0.1"), Pair.of("dbms.shell.port", "1337") ); // ^^ serverBootstrapper.start() also registered shutdown hook! NeoServer neoServer = serverBootstrapper.getServer(); GraphDatabaseService gdb = neoServer.getDatabase().getGraph(); /* Some transactional code */ try(Transaction tx = gdb.beginTx()) { gdb.getAllNodes().forEach( n -> System.out.println(n) ); tx.success(); } System.out.println("Press ENTER to quit."); System.in.read(); System.exit(0); } }