I read from a table in Oracle and insert the whole dump in Db2. The table structures are the same. I am using the Simple scala class, which performs the above task. I set the batchsize inserts to be 300. After several batches are updated, the class throws an exception below
com.ibm.db2.jcc.am.SqlIntegrityConstraintViolationException: Error for batch element #10: DB2 SQL Error: SQLCODE=-803, SQLSTATE=23505, SQLERRMC=1;PME.TM_ASSET_LQA_DETL, DRIVER=4.13.127
at com.ibm.db2.jcc.am.id.a(id.java:673) ~[db2jcc-4.13.127.jar:na]
at com.ibm.db2.jcc.am.id.a(id.java:60) ~[db2jcc-4.13.127.jar:na]
at com.ibm.db2.jcc.am.id.a(id.java:127) ~[db2jcc-4.13.127.jar:na]
at com.ibm.db2.jcc.t4.cb.a(cb.java:481) ~[db2jcc-4.13.127.jar:na]
at com.ibm.db2.jcc.t4.cb.a(cb.java:70) ~[db2jcc-4.13.127.jar:na]
at com.ibm.db2.jcc.t4.q.a(q.java:57) ~[db2jcc-4.13.127.jar:na]
at com.ibm.db2.jcc.t4.tb.a(tb.java:225) ~[db2jcc-4.13.127.jar:na]
at com.ibm.db2.jcc.am.oo.a(oo.java:3434) ~[db2jcc-4.13.127.jar:na]
at com.ibm.db2.jcc.am.oo.d(oo.java:5550) ~[db2jcc-4.13.127.jar:na]
at com.ibm.db2.jcc.am.oo.a(oo.java:4992) ~[db2jcc-4.13.127.jar:na]
at com.ibm.db2.jcc.am.oo.c(oo.java:4664) ~[db2jcc-4.13.127.jar:na]
at com.ibm.db2.jcc.am.oo.executeBatch(oo.java:2934) ~[db2jcc-4.13.127.jar:na]
at com.baml.regw.db.replicator.ReplicationRunnable$$anonfun$run$3.apply(SimpleReplicator.scala:105) ~[regw-db-replicator-0.0.933-SNAPSHOT.jar:na]
at com.baml.regw.db.replicator.ReplicationRunnable$$anonfun$run$3.apply(SimpleReplicator.scala:80) ~[regw-db-replicator-0.0.933-SNAPSHOT.jar:na]
Since the exception was due to IntegrityConstraint, I tried to check for a primary primary key (ID + TimeStamp + 9999-12-31 00.00.000000), but the combo is not present in either the Oracle table or the Db2 table. Limitations for the Db2 table:
COLUMN NAME UNIQUE RULE
+ID+BUSINESS_STOP+BUSINESS_START Primary
+ID Duplicate
+BUSINESS_START Duplicate
+LOW_QUALITY_IND Duplicate
+IDENTIFIER1 Duplicate
+IDENTIFIER2 Duplicate
I looked at other similar issues in SO, but none of the fixes worked for me. The code that performs this task
logger.info("Retrieving based on query string: " + queryStr + " for thread " + threadNum)
val start = System.currentTimeMillis()
val rs = stmt.executeQuery(queryStr)
val rsMd = rs.getMetaData()
val end = System.currentTimeMillis()
logger.info("Query execution time: " + (end - start) + "ms.")
done = true
var stmtCount = 0
Iterator.continually(rs).takeWhile(_.next()).foreach { rs =>
if (sourceConf.hasPath("blockSize")) {
done = false
}
for (idx <- 1 to (rsMd.getColumnCount()-extraColumnCount)) {
try {
logger.info("destStmt.setObject"+rs.getObject(idx)+" column Type "+ rsMd.getColumnType(idx))
destStmt.setObject(idx, rs.getObject(idx), rsMd.getColumnType(idx))
}
catch {
case e:Exception => {
logger.warn("While attempting to set (1-based) index: " + idx +
" to value of type " + {if(rs.getObject(idx) != null) rs.getObject(idx).getClass().getName() else "[NULL]"} +
" received error: " + e.getMessage())
throw e
}
}
}
destStmt.addBatch()
stmtCount += 1
if(stmtCount % { if (destConf.hasPath("batchSize")) destConf.getInt("batchSize") else 200 } == 0) {
destStmt.executeBatch()
destDbConn.commit()
destStmt.clearBatch()
stmtCount = 0
}
}
if(stmtCount > 0) {
destStmt.executeBatch()
destDbConn.commit()
destStmt.clearBatch()
}
rs.close()
stmt.close()