How can I use JavaFX to play videos in 3D form?

Question

I am new to JavaFX and Java. I am wondering how to make a scene on the surface of a cube or any polyhedrons? I would like to play the video on the surface of any 3D shape. How can i do this?

Answer 1

Here is an example that uses the built-in JavaFX MediaPlayer and takes periodic snapshots of the presentation on the media in a texture image displayed on a three-dimensional shape (in this case, a box). The rotation animation around the Y axis is added so that the sides of the box can be seen in perspective.

import javafx.animation.*; import javafx.application.*; import javafx.geometry.Rectangle2D; import javafx.scene.*; import javafx.scene.image.WritableImage; import javafx.scene.media.*; import javafx.scene.paint.*; import javafx.scene.shape.*; import javafx.scene.transform.Rotate; import javafx.stage.Stage; import javafx.util.Duration; // Display a rotating 3D box with a video projected onto its surface. public class ThreeDMedia extends Application { private static final String MEDIA_URL = "http://download.oracle.com/otndocs/products/javafx/oow2010-2.flv"; private static final int SCENE_W = 640; private static final int SCENE_H = 400; private static final double MEDIA_W = 540 * 2/3; private static final double MEDIA_H = 209 * 2/3; private static final Color INDIA_INK = Color.rgb(35, 39, 50); @Override public void start(Stage stage) { // create a 3D box shape on which to project the video. Box box = new Box(MEDIA_W, MEDIA_H, MEDIA_W); box.setTranslateX(SCENE_W / 2); box.setTranslateY(SCENE_H / 2); // create a media player for the video which loops the video forever. MediaPlayer player = new MediaPlayer(new Media(MEDIA_URL)); player.setCycleCount(MediaPlayer.INDEFINITE); // create a media view for the video, sized to our specifications. MediaView mediaView = new MediaView(player); mediaView.setPreserveRatio(false); mediaView.setFitWidth(MEDIA_W); mediaView.setFitHeight(MEDIA_H); // project the video on to the 3D box. showMediaOnShape3D(box, mediaView); // rotate the box. rotateAroundYAxis(box); // create a point light source a fair way away so lighting is reasonably even. PointLight pointLight = new PointLight( Color.WHITE ); pointLight.setTranslateX(SCENE_W / 2); pointLight.setTranslateY(SCENE_H / 2); pointLight.setTranslateZ(-SCENE_W * 5); // add a bit of ambient light to make the lighting more natural. AmbientLight ambientLight = new AmbientLight( Color.rgb(15, 15, 15) ); // place the shape and associated lights in a group. Group group = new Group( box, pointLight, ambientLight ); // create a 3D scene with a default perspective camera. Scene scene = new Scene( group, SCENE_W, SCENE_H, true, SceneAntialiasing.BALANCED ); scene.setFill(INDIA_INK); PerspectiveCamera camera = new PerspectiveCamera(); scene.setCamera(camera); stage.setScene(scene); stage.setResizable(false); // start playing the media, showing the scene once the media is ready to play. player.setOnReady(stage::show); player.setOnError(Platform::exit); player.play(); } // Project video on to 3D shape. private void showMediaOnShape3D(Shape3D shape3D, final MediaView mediaView) { PhongMaterial material = new PhongMaterial(); shape3D.setMaterial(material); Scene mediaScene = new Scene( new Group(mediaView), MEDIA_W, MEDIA_H ); SnapshotParameters snapshotParameters = new SnapshotParameters(); snapshotParameters.setViewport( new Rectangle2D( 0, 0, MEDIA_W, MEDIA_H ) ); WritableImage textureImage = mediaView.snapshot( snapshotParameters, null ); material.setDiffuseMap(textureImage); AnimationTimer timer = new AnimationTimer() { @Override public void handle(long now) { mediaView.snapshot( snapshotParameters, textureImage ); } }; timer.start(); } // Rotates a shape around the y axis indefinitely. private void rotateAroundYAxis(Shape3D shape3D) { RotateTransition rotateY = new RotateTransition( Duration.seconds(10), shape3D ); rotateY.setAxis(Rotate.Y_AXIS); rotateY.setFromAngle(360); rotateY.setToAngle(0); rotateY.setCycleCount(RotateTransition.INDEFINITE); rotateY.setInterpolator(Interpolator.LINEAR); rotateY.play(); } public static void main(String[] args) { launch(args); } } 

Answer 2

Actually, you can play the video in any 3D form.

There's a great project for playing videos with @caprica called VLCJ : Java Infrastructure for VLC Media Player .

While the project is intended for rendering in AWT canvas, the author made several tests to display it also in JavaFX Canvas .

Based on its JavaFX class, it is easy to display a buffer in a 3D form instead of a 2D node canvas.

Settings

First you need to install the first VLC video player.

Then you will need some dependencies: vlcj-3.6.0.jar , jna-3.5-2.jar and platform-3.5.2.jar and slfj4j-api.1.7.12.jar.

In addition, I will use some custom 3D forms from FXyz , although you can use the regular ones from the API, like a Box .

The basics

The trick for rendering video in 3D form uses a diffuse map of its material, which takes an image and determines its texture.

You can find more information about this here or here .

So, for each available frame, we will create a new image and set it as a diffuse map:

 ByteBuffer byteBuffer = nativeBuffer.getByteBuffer(0, nativeBuffer.size()); BufferFormat bufferFormat = ((DefaultDirectMediaPlayer) mediaPlayerComponent.getMediaPlayer()).getBufferFormat(); WritableImage textureImage = new WritableImage(bufferFormat.getWidth(), bufferFormat.getHeight()); if (bufferFormat.getWidth() > 0 && bufferFormat.getHeight() > 0) { textureImage.getPixelWriter().setPixels(0, 0, bufferFormat.getWidth(), bufferFormat.getHeight(), pixelFormat, byteBuffer, bufferFormat.getPitches()[0]); // apply new frame as texture image to the 3D shape material material.setDiffuseMap(textureImage); } 

An AnimationTimer allows you to update frames and textures.

Example

This is a working sample that displays video on a segmented torus.

 public class Video3D extends Application { static { // path to the VLC video player System.setProperty("jna.library.path", "C:/Program Files/VideoLAN/VLC"); } // http://download.blender.org/peach/bigbuckbunny_movies/ // (c) copyright 2008, Blender Foundation / www.bigbuckbunny.org private static final String VIDEO_FILE = "C:\\BigBuckBunny_320x180.mp4"; private final DirectMediaPlayerComponent mediaPlayerComponent; private final WritablePixelFormat<ByteBuffer> pixelFormat; private final SegmentedTorusMesh torus = new SegmentedTorusMesh(50,40,12,3.2d,4.5d); private final PhongMaterial material = new PhongMaterial(Color.WHEAT); private double mousePosX, mousePosY; private double mouseOldX, mouseOldY; private final Rotate rotateX = new Rotate(-20, Rotate.X_AXIS); private final Rotate rotateY = new Rotate(240, Rotate.Y_AXIS); private final AnimationTimer timer; public TestVLC(){ mediaPlayerComponent = new TestMediaPlayerComponent(); pixelFormat = PixelFormat.getByteBgraInstance(); timer = new AnimationTimer() { @Override public void handle(long now) { renderFrame(); } }; } protected void startTimer() { mediaPlayerComponent.getMediaPlayer().playMedia(VIDEO_FILE); timer.start(); } protected void stopTimer() { mediaPlayerComponent.getMediaPlayer().stop(); timer.stop(); } @Override public void start(Stage primaryStage) { torus.setCullFace(CullFace.NONE); torus.setzOffset(1.4); torus.setMaterial(material); PerspectiveCamera camera = new PerspectiveCamera(true); camera.getTransforms().addAll (rotateX, rotateY, new Translate(0, 0, -30)); Group root3D = new Group(camera,torus); SubScene subScene = new SubScene(root3D, 800, 600, true, SceneAntialiasing.BALANCED); subScene.setFill(Color.AQUAMARINE); subScene.setCamera(camera); BorderPane pane = new BorderPane(); pane.setCenter(subScene); Button play = new Button("Play"); play.setOnAction(e->startTimer()); Button stop = new Button("Stop"); stop.setOnAction(e->stopTimer()); ToolBar toolBar = new ToolBar(play, stop); toolBar.setOrientation(Orientation.VERTICAL); pane.setRight(toolBar); pane.setPrefSize(600,400); Scene scene = new Scene(pane); scene.setOnMousePressed((MouseEvent me) -> { mouseOldX = me.getSceneX(); mouseOldY = me.getSceneY(); }); scene.setOnMouseDragged((MouseEvent me) -> { mousePosX = me.getSceneX(); mousePosY = me.getSceneY(); rotateX.setAngle(rotateX.getAngle()-(mousePosY - mouseOldY)); rotateY.setAngle(rotateY.getAngle()+(mousePosX - mouseOldX)); mouseOldX = mousePosX; mouseOldY = mousePosY; }); primaryStage.setScene(scene); primaryStage.setTitle("Video - JavaFX 3D"); primaryStage.show(); } @Override public final void stop() throws Exception { stopTimer(); mediaPlayerComponent.getMediaPlayer().stop(); mediaPlayerComponent.getMediaPlayer().release(); } /** * Implementation of a direct rendering media player component that renders * the video to a JavaFX canvas. * https://github.com/caprica/vlcj-javafx/blob/master/src/test/java/uk/co/caprica/vlcj/javafx/test/JavaFXDirectRenderingTest.java */ private class TestMediaPlayerComponent extends DirectMediaPlayerComponent { public TestMediaPlayerComponent() { super(new TestBufferFormatCallback()); } } /** * Callback to get the buffer format to use for video playback. */ private class TestBufferFormatCallback implements BufferFormatCallback { @Override public BufferFormat getBufferFormat(int sourceWidth, int sourceHeight) { final int width = sourceWidth; final int height = sourceHeight; Platform.runLater(() -> { torus.setMajorRadius(width/100); torus.setMinorRadius(height/40); }); return new RV32BufferFormat(width, height); } } protected final void renderFrame() { Memory[] nativeBuffers = mediaPlayerComponent.getMediaPlayer().lock(); if (nativeBuffers != null) { Memory nativeBuffer = nativeBuffers[0]; if (nativeBuffer != null) { ByteBuffer byteBuffer = nativeBuffer.getByteBuffer(0, nativeBuffer.size()); BufferFormat bufferFormat = ((DefaultDirectMediaPlayer) mediaPlayerComponent.getMediaPlayer()).getBufferFormat(); WritableImage textureImage = new WritableImage(bufferFormat.getWidth(), bufferFormat.getHeight()); if (bufferFormat.getWidth() > 0 && bufferFormat.getHeight() > 0) { textureImage.getPixelWriter().setPixels(0, 0, bufferFormat.getWidth(), bufferFormat.getHeight(), pixelFormat, byteBuffer, bufferFormat.getPitches()[0]); material.setDiffuseMap(textureImage); } } } mediaPlayerComponent.getMediaPlayer().unlock(); } public static void main(String[] args) { launch(args); } } 

And these are just two snapshots of what you get.

Answer 3

In addition to jewelsea and José more elegant solutions, you can always position MediaView manually in the 3d space e. g. as 6 faces of a cube:

 public class VideoCubeDemo extends Application { Random rnd = new Random(); // size of the cube double size = 320; @Override public void start(Stage primaryStage) throws MalformedURLException { // create media views List<String> videoFiles = new ArrayList<>(); videoFiles.add( getClass().getResource("funny_cats_compilation_2012.mp4").toExternalForm()); videoFiles.add( getClass().getResource("funny_cats_compilation_2012.mp4").toExternalForm()); videoFiles.add( getClass().getResource("funny_cats_compilation_2012.mp4").toExternalForm()); videoFiles.add( getClass().getResource("funny_cats_compilation_2012.mp4").toExternalForm()); videoFiles.add( getClass().getResource("funny_cats_compilation_2012.mp4").toExternalForm()); videoFiles.add( getClass().getResource("funny_cats_compilation_2012.mp4").toExternalForm()); // create faces for the cube // original cube face code from http://www.javafxapps.in/tutorial/Creating-3D-Cube-in-javafx.html MediaView r; int videoIndex; Group cube = new Group(); List<MediaView> cubeFaces = new ArrayList<>(); // back face videoIndex = 0; r = createMediaView(videoFiles.get( videoIndex)); r.setTranslateX(-0.5 * size); r.setTranslateY(-0.5 * size); r.setTranslateZ(0.5 * size); cubeFaces.add( r); // bottom face videoIndex = 1; r = createMediaView(videoFiles.get( videoIndex)); r.setTranslateX(-0.5 * size); r.setTranslateY(0); r.setRotationAxis(Rotate.X_AXIS); r.setRotate(90); cubeFaces.add( r); // right face videoIndex = 2; r = createMediaView(videoFiles.get( videoIndex)); r.setTranslateX(-1 * size); r.setTranslateY(-0.5 * size); r.setRotationAxis(Rotate.Y_AXIS); r.setRotate(90); cubeFaces.add( r); // left face videoIndex = 3; r = createMediaView(videoFiles.get( videoIndex)); r.setTranslateX(0); r.setTranslateY(-0.5 * size); r.setRotationAxis(Rotate.Y_AXIS); r.setRotate(90); cubeFaces.add( r); // top face videoIndex = 4; r = createMediaView(videoFiles.get( videoIndex)); r.setTranslateX(-0.5 * size); r.setTranslateY(-1 * size); r.setRotationAxis(Rotate.X_AXIS); r.setRotate(90); cubeFaces.add( r); // front face videoIndex = 5; r = createMediaView(videoFiles.get( videoIndex)); r.setTranslateX(-0.5 * size); r.setTranslateY(-0.5 * size); r.setTranslateZ(-0.5 * size); cubeFaces.add( r); // create cube with all faces cube.getChildren().addAll( cubeFaces); // initial cube rotation cube.getTransforms().addAll(new Rotate(45, Rotate.X_AXIS), new Rotate(45, Rotate.Y_AXIS)); // animate cube Point3D rotateAxis = new Point3D(1,1,1); // rotate around X, Y and Z Timeline animation = new Timeline(); animation.getKeyFrames().addAll( new KeyFrame(Duration.ZERO, new KeyValue(cube.rotationAxisProperty(), rotateAxis), new KeyValue(cube.rotateProperty(), 0d)), new KeyFrame(Duration.seconds(5), new KeyValue(cube.rotationAxisProperty(), rotateAxis), new KeyValue(cube.rotateProperty(), 360d)) ); animation.setCycleCount(Animation.INDEFINITE); // add objects to scene StackPane root = new StackPane(); root.getChildren().add(cube); Scene scene = new Scene(root, 1600, 900, true, SceneAntialiasing.BALANCED); scene.setFill(Color.BLACK); scene.setCamera(new PerspectiveCamera()); primaryStage.setResizable(true); primaryStage.setScene(scene); primaryStage.show(); // play videos and animation for( MediaView mediaPlayer: cubeFaces) { mediaPlayer.getMediaPlayer().play(); } animation.play(); } private MediaView createMediaView( String path) { Media media = new Media( path); MediaPlayer mediaPlayer = new MediaPlayer( media); mediaPlayer.setVolume(0.8); mediaPlayer.setCycleCount(MediaPlayer.INDEFINITE); MediaView mediaView = new MediaView( mediaPlayer); mediaView.setFitHeight(size); mediaView.setFitWidth(size); mediaView.setPreserveRatio( false); return mediaView; } public static void main(String[] args) { launch(args); } }