In this work, we tackle the problem of indoor scene understanding using RGBD data. We propose a holistic approach that exploits segmentation, 3D geometry, as well as contextual relations between scenes and objects. Specifically, we extend the CPMC [Carreira et al., 2012] framework to 3D in order to generate candidate cuboids, and develop a conditional random field to integrate information from different sources to classify the cuboids. With this formulation, scene classification and 3D object recognition are coupled and can be jointly solved through probabilistic inference. We test the effectiveness of our approach on the challenging NYU v2 dataset. The experimental results demonstrate that our approach achieves substantial improvement over the state-of-the-art.