Geek Answers Handbook

REST (Squeryl / Akka / Spray) - very low bandwidth

I am currently creating my first REST API based on the RSS aggregator. I implemented it using one of two characteristics: either MemoryBasedDB or PostgresDB. Each time the root URL is accessed, it will call the feed asynchronously to capture the latest articles, and return this as an XML string for parsing. After parsing, it is saved in the database as an Article object.

Functionally, this is great for me. However, when testing the load using weighttp or gatling, it will fail on 1k / 1k requests that use Postgres with the following:

In weighttp:

error: read() failed: Connection reset by peer (104)

And in my server logs:

final [WARN] [09/21/2014 14:45:27.224] [on-spray-can-akka.actor.default-dispatcher-36] [akka://on-spray-can/user/IO-HTTP/listener-0/523] Configured registration timeout of 1 second expired, stopping

I believe that this is due to the way my requests are laid out. They are blocked, and since each actor must wait for an answer, the load behind them accumulates higher and higher until failure (timeout). However, in my research, I was able to find this asynchronous driver for postgres, which is currently not compatible with Squeryl (as far as I know).

How can I make access to the database faster? Currently I am reaching ~ 10-15req / s using Postgres and ~ 400req / s using memory storage.

My model:

case class Article(id: Option[String], idint: Option[Int], title: String, author: String, published: String, updated: String, `abstract`: Option[String], content: Option[String], link: Option[String])

My queries are:

trait PostgresDB extends Schema {

  val articles = table[Article]("articles")
  on(articles)(e => declare(e.idint is(unique)))

  def create(x: Article) = inTransaction {
      articles.insert(x)
  }

  def getAll: Set[Article] = inTransaction {
      from(articles)(article => select(article)).toSet
  }

  def getArticle(x: Int) = inTransaction {
      from(articles)(article => where(article.idint === Some(x)) select(article)).toList(0)
  }

  def printy = transaction {
      articles.schema.printDdl(println(_))
  }
}

So far I have tried:

  • Implement C3P0 for pooling. No real changes.
  • Tuning postgresql.conf for performance. A slight positive change.
  • Configure .conf application for atomization / acc to run. A slight positive change.

Relevant Information:

  • Core:
    • Linux 3.13.0-33-generi# 58-Ubuntu SMP 29 16:45:05 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
  • Postgres 9.3
  • Scala 2.10.4
  • Spray 1.3.1
  • Akka 2.3.5
+4
3

. @experquiste, db , db. . . , , .

db. , , . http://www.chrisstucchio.com/blog/2013/actors_vs_futures.html concurrency.

squeryl, inTransaction db? , , , , .

+3

, REST db-... db ?

+3

, ? , , .

. "~ 10-15req/s" , . .

. , C3P0 . , ?

I also suggest being careful with asynchronous code and thread pools, avoiding them if they are not needed. They make the code more complex and open up a whole new field for errors. (Async is still cool in some use cases, but I hope I already made that clear.)

0
source

More articles:


All Articles