Capturing from the screen and saving to the multithreaded disk

Question

The next task is to monitor the screen, record the event (the measurement text field turns green) and record all the events leading to it, creating a “movie” of the events leading to it. Unfortunately, the entire screen needs to be recorded. I have so far fulfilled the role in which recognition takes part. However, I barely get two frames per second. I would like to have about 25 to 30 fps .

My idea was to write and read in two separate streams. Because a recording event is rare and can be performed in the background , a recording event may take longer and run faster. Unfortunately, all this seems too slow. I would like to be able to write a screen to disk at intervals of 10 to 20 seconds before .

Edit: If possible, I would like to remain as platform independent as possible.

Edit 2: it seems that for Xuggler there is a platform-independent jar file. Unfortunately, I really don't understand how I can use it for my purpose: recording 20 seconds until the moment when isarecord starts.

Here is what I have done so far:

package fragrecord; import java.awt.AWTException; import java.awt.Color; import java.awt.Dimension; import java.awt.Rectangle; import java.awt.Robot; import java.awt.image.BufferedImage; import java.io.File; import java.io.IOException; import java.util.concurrent.ExecutorService; import java.util.concurrent.Executors; import java.util.concurrent.LinkedBlockingQueue; import java.util.concurrent.ThreadPoolExecutor; import java.util.concurrent.TimeUnit; import javax.imageio.ImageIO; public class Main { public static void main(String[] args) { //The numbers are just silly tune parameters. Refer to the API. //The important thing is, we are passing a bounded queue. ExecutorService consumer = new ThreadPoolExecutor(1,4,30,TimeUnit.SECONDS,new LinkedBlockingQueue<Runnable>(100)); System.out.println("starting"); //No need to bound the queue for this executor. //Use utility method instead of the complicated Constructor. ExecutorService producer = Executors.newSingleThreadExecutor(); Runnable produce = new Produce(consumer); producer.submit(produce); try { producer.awaitTermination(Long.MAX_VALUE, TimeUnit.NANOSECONDS); } catch (InterruptedException e) { // TODO Auto-generated catch block e.printStackTrace(); } consumer.shutdown(); try { consumer.awaitTermination(Long.MAX_VALUE, TimeUnit.NANOSECONDS); } catch (InterruptedException e) { // TODO Auto-generated catch block e.printStackTrace(); } } } class Produce implements Runnable { private final ExecutorService consumer; public Produce(ExecutorService consumer) { this.consumer = consumer; } boolean isarecord(BufferedImage image){ int x=10, y = 10; Color c = new Color(image.getRGB(x,y)); int red = c.getRed(); int green = c.getGreen(); int blue = c.getBlue(); // Determine whether to start recording return false; } @Override public void run() { Robot robot = null; try { robot = new Robot(); } catch (AWTException e) { // TODO Auto-generated catch block e.printStackTrace(); } // // Capture screen from the top left to bottom right // int i = 0; while(true) { i++; BufferedImage bufferedImage = robot.createScreenCapture( new Rectangle(new Dimension(1024, 798))); Runnable consume = new Consume(bufferedImage,i); consumer.submit(consume); } } } class Consume implements Runnable { private final BufferedImage bufferedImage; private final Integer picnr; public Consume(BufferedImage bufferedImage, Integer picnr){ this.bufferedImage = bufferedImage; this.picnr = picnr; } @Override public void run() { File imageFile = new File("screenshot"+picnr+".png"); try { System.out.println("screenshot"+picnr+".png"); ImageIO.write(bufferedImage, "png", imageFile); } catch (IOException e) { // TODO Auto-generated catch block e.printStackTrace(); } } } 

Answer 1

for various reasons using artists, rewriting on nio and similar things will not help.

here are a bunch of things you should consider:

• Capturing a SLOW image in java, you can't do anything with it without JNI dependencies.
• use jpeg instead of png, much faster
• any SLOW image compression using ImageIO. you can use old-school classes, the proprietary JpegEncoder classes (in the com.sun package) or the TurboJPEG java library (JNI also), but this is not a bottleneck
• disk i / o DEFINITELY not a problem (you write <5mb / s, so don't worry)
• Recording multiple images in parallel will slow down your application, not speed it up (if you don't have ssd)
• consider parallelizing the capture / analysis flow instead (for example, analyze only every 20th frame) (*)
• I am sure you would finish writing this application twice for each platform using your native language before you can optimize your java application to run at 25 frames per second (using a 100% xD processor).
• seriously consider hybrid solutions; for example, record everything in a compressed movie using an available tool, then do the analysis later (**)

in a word: java sucks what you want to do is really bad.

However, I wrote my own version of this tool. http://pastebin.com/5h285fQw

one thing he does is write a smaller rectangle following the mouse. at 500x500 it easily changes to 25 frames per second when images are recorded in the background (image compression + recording takes 5-10 ms for me, so it is recorded much faster than recordings)

---

(*) you are not talking about how you analyze images, but this seems to be the main source of your performance problem. some ideas:

• Look at every Nth frame only.
• take a capture of only a subsection of the screen and move this part over time over time (model the images later, this will cause a terrible break, but maybe it doesn’t matter for your purpose).
• capture should be "not too slow" (maybe 10-20 frames per second for full screen mode); use sequential capture, but parallelize analysis.

(**) on macosx you can use the built-in QuickTime x to write to hdd quite efficiently; on the windows I heard that playclaw (http://www.playclaw.com/) is not bad and may be worth the money (think what would you do in the wasted time optimizing java beast :))

Answer 2

I tried changing your code a bit. Instead of creating png files, I tried to create bmp files that remove the overhead of data compression, but at the cost of disk space.

Result: I do not know how to read fps, but the solution is faster than yours. :-)

Answer 3

You need to measure how long robot.createScreenCapture() takes. Most likely, it needs more than 40 ms, which means that there is no way to achieve what you want with pure Java. In my experience, the challenge can be very slow.

I managed to significantly reduce this point with the trick, but it only works on Unix: start the VNC server (= desktop in RAM). I fixed the source code of TightVNC to use NIO to write images to disk using a memory mapped file. This gave me about 10-20 frames per second.

Here is the code for recording an image using NIO:

 private File pixelFile = new File("tmp", "pixels.nio").getAbsoluteFile(); private IntBuffer intBuffer; private FileChannel rwChannel; private MappedByteBuffer byteBuffer; private int[] pixels; private void createMemoryMappedFile() { File dir = pixelFile.getParentFile(); if(!dir.exists()) { dir.mkdirs(); } try { rwChannel = new RandomAccessFile(pixelFile, "rw").getChannel(); int width = ...; int height = ...; pixels = new int[width*height]; byteBuffer = rwChannel.map(MapMode.READ_WRITE, 0, width * height * 4); intBuffer = byteBuffer.asIntBuffer(); } catch(Exception e) { throw new RuntimeException("Error creating NIO file " + pixelFile, e); } } public void saveImage() { buffer.position(0); buffer.put(image.getRaster().getPixels(0,0,width,height,pixels)); flushPixels(); } private void flushPixels() { byteBuffer.force(); } 

Answer 4

The biggest problem is that you only get one stream to create images. ThreadPoolExecutor does not create threads as you expect.

From javadoc

• If fewer corePoolSize threads are running, Executor always prefers adding a new thread rather than queuing.
• If corePoolSize or more threads are running, Executor always prefers to post a request rather than add a new thread.
• If the request cannot be queued, a new thread is created if it does not exceed maximumPoolSize, in which case the task will be rejected.

Thus, it will use only one thread if the queue is not full. At this time, you have 100 screenshots in memory that add work to the GC. If I set the main threads to 4 (I have 4 cores on my laptop) and increase the memory to 1 GB, I managed to capture 20 FPS or so.

If your output to disk is limited, you can save the last 400 recorded images as arrays of bytes in the queue and only write them to disk when the button turns green. However, in my tests, these images took up more than 100 MB of memory, so once again make sure that you have enough memory.

Answer 5

Disable the theme, but check out Xuggler . It would be useful if you want to create a video with Java.

Change In addition, you can optimize your consumer image code if you do not upload each image to disk, but add it to an array of bytes and rarely upload it to disk.

Editing 2: there is a “no install" version of the library and a maven dependency for it (jar with precompiled libraries for the platform): blog.xuggle.com/2012/04/03/death-to-installers and xuggle.com/xuggler/ downloads