diffrp.rendering.surface_deferred#

class diffrp.rendering.surface_deferred.SurfaceDeferredRenderSession(scene: Scene, camera: Camera, opaque_only: bool = True, options: SurfaceDeferredRenderSessionOptions | None = None, ctx: RasterizeCudaContext | RasterizeGLContext | None = None)#

Bases: RenderSessionMixin

The heart of the deferred rasterization render pipeline in DiffRP. Most intermediate options in this class are cached, so you are recommended to create a new session whenever you change the scene and/or the camera, including rendering multiple frames of the same object, or changing attributes on the object. Initialization of this class is made light-weight and fast.

It is thread-safe to run more than one sessions on the same GPU by default.

DiffRP operations do not support multiple GPUs on a single process. You will need to access the GPUs with a distributed paradigm, where each process only uses one GPU.

Parameters:

  • scene (Scene) – The scene to be rendered.

  • camera (Camera) – The camera to be rendered.

  • opaque_only (bool) – Render in opaque-only mode. Drops support for transparency, and forces all alpha values to be 1. This can save memory and time if you do not need transparency support. Defaults to True. You need to specify False if you need transparency.

  • options (SurfaceDeferredRenderSessionOptions) – Options for the session. See SurfaceDeferredRenderSessionOptions for details.

  • ctx (RasterizeContext) – The backend context to be used. By default, each GPU and each thread has its own, shared context. This render pipeline uses nvdiffrast as the backend. You can pass your own RasterizeCudaContext or RasterizeGLContext if needed.

__init__(scene: Scene, camera: Camera, opaque_only: bool = True, options: SurfaceDeferredRenderSessionOptions | None = None, ctx: RasterizeCudaContext | RasterizeGLContext | None = None) → None#
albedo()#
Returns:

Rendered albedo, or base color RGBA, in linear space. Albedo from transparent objects are alpha-blended if opaque_only is disabled. Tensor of shape (H, W, 4).

Return type:

torch.Tensor

albedo_layered() → List[Tensor]#
albedo_srgb()#
Returns:

Rendered albedo, or base color RGBA. RGBs are in sRGB space. Albedo from transparent objects are alpha-blended in linera space if opaque_only is disabled. Tensor of shape (H, W, 4).

Return type:

torch.Tensor

alpha_layered() → List[Tensor]#
Returns:

Alphas of layers in the current scene.

Return type:

List[torch.Tensor]

aov(key: str, bg_value: List[float])#
Parameters:

  • key (str) – The key you want to collect from material AOV outputs.

  • bg_value (List[float]) – Values to fill in where the AOV is either not specified in materials or background is hit.

Returns:

Collected full-screen buffers of AOVs. Values are alpha-blended for transparent objects, if opaque_only is disabled. Alpha channel is geometry transparency. Tensor of shape (H, W, C + 1).

Return type:

torch.Tensor

See also

diffrp.materials.base_material.SurfaceOutputStandard

camera_space_normal()#
Returns:

Camera-space normals ([0, 0, 1] is directly facing the camera). Raw values of unit vectors. Values are alpha-blended for transparent objects, if opaque_only is disabled. Alpha channel is geometry transparency. Tensor of shape (H, W, 4).

Return type:

torch.Tensor

camera_space_normal_layered() → List[Tensor]#
clip_space()#

All vertices concatenated, transformed into the GL clip space for projection.

Returns:

Tensor of shape (V, 4).

Return type:

torch.Tensor

compose_layers(colors: list, alphas: list | None = None, blend_fn: Callable | None = None, return_alpha: bool = True) → Tensor#

Compose multiple layers into one. The layers in the front in the lists are the layers in the front in 3D.

Parameters:

  • colors (List[torch.Tensor]) – RGB colors, or arbitrary values to be blended. Tensors can have arbitrary shapes, but consistent across the list.

  • alphas (List[torch.Tensor]) – Alphas. Tensors should have matching shapes with colors, with the last dimension (channels) equal to 1. Defaults to alphas of layers in the current scene.

  • blend_fn (callable) – A function that accepts 4 arguments: (fg_value, fg_alpha, bg_value, bg_alpha) and returns a tuple of composition results (value, alpha). Defaults to alpha blending.

  • return_alpha (bool) – Whether to include the alpha channel in the result. The alpha channel will be concatenated to the end if set.

depth()#
Returns:

Depth of the scene, raw values. Depth is distance to the camera-z plane. Values are alpha-blended for transparent objects, if opaque_only is disabled. Emptiness results in zero depth. Tensor of shape (H, W, 1).

Return type:

torch.Tensor

depth_layered() → List[Tensor]#
distance()#
Returns:

Distance of the scene to the camera, raw values. Has a similar effect but slightly different definition to depth. Values are alpha-blended for transparent objects, if opaque_only is disabled. Emptiness results in zero distance. Tensor of shape (H, W, 1).

Return type:

torch.Tensor

distance_layered() → List[Tensor]#
edge_gradient(rgb: Tensor)#

Applies the edge-gradient technique in Rasterized Edge Gradients: Handling Discontinuities Differentiably. This can generate visibility gradients in differentiable rendering.

The image will be returned as-is in the forward session, but gradients can start flowing through geometry visibility.

Semi-transparent objects are not well-supported by this operation.

Parameters:

rgb (torch.Tensor) – Tensor of shape (H, W, 3). RGB image to compute edge gradient with. Must be rendered in this session.

Returns:

the same RGB image, but with gradient operators attached.

Return type:

torch.Tensor

emission()#
Returns:

Rendered direct emission RGB, in linear space. Emission is composed additively for transparent objects if opaque_only is disabled. Tensor of shape (H, W, 3).

Return type:

torch.Tensor

emission_layered() → List[Tensor]#
false_color_camera_space_normal()#
Returns:

Camera-space normals XYZ linearly mapped to RGB space from [-1, 1] to [0, 1], the common blue-ish rendering of scene normals. Values are alpha-blended for transparent objects, if opaque_only is disabled. Alpha channel is geometry transparency. Tensor of shape (H, W, 4).

Return type:

torch.Tensor

false_color_mask_mso()#
Returns:

Rendered Metallic, Smoothness and ambiend Occlusion values linearly mapped to RGB channels. Values are alpha-blended for transparent objects, if opaque_only is disabled. Alpha channel is geometry transparency. Tensor of shape (H, W, 4).

Return type:

torch.Tensor

false_color_nocs()#

Note: Objects are NOT automatically normalized before this rendering!

Returns:

Object-space positions XYZ linearly mapped to RGB space from [-1, 1] to [0, 1], conformal with the NOCS definition. Values are alpha-blended for transparent objects, if opaque_only is disabled. Alpha channel is geometry transparency. Tensor of shape (H, W, 4).

Return type:

torch.Tensor

false_color_world_space_normal()#
Returns:

World-space normals XYZ linearly mapped to RGB space from [-1, 1] to [0, 1]. Values are alpha-blended for transparent objects, if opaque_only is disabled. Alpha channel is geometry transparency. Tensor of shape (H, W, 4).

Return type:

torch.Tensor

gbuffer_collect(operator: Callable[[SurfaceInput, SurfaceOutputStandard], Tensor], default: List[float] | Tensor) → List[Tensor]#

Collect the screen G-buffers from material evaluation results.

The results from the operator given are collected against all materials into screen-sized buffers, one per rasterized layer.

Parameters:

  • operator (callable) – A function that takes two arguments, SurfaceInput and SurfaceOutputStandard as input, and outputs a tensor.

  • default (List[float]) – If the operator returns None, the default value is filled for the material. The default value is also used for inactive pixels (pixels without geometry).

Returns:

Layered G-buffers collected.

Return type:

List[torch.Tensor]

ibl(world_normal: Tensor, view_dir: Tensor, mso: Tensor, albedo: Tensor)#

Evaluate Image-based Lighting (IBL) given the necessary G-buffers.

Uses split-sum approximation and pre-filtering techniques.

layer_material() → List[List[Tuple[SurfaceInput, SurfaceOutputStandard]]]#

Evaluate materials for the scene. Each layer is a list of tensors of material inputs/outputs. Shapes of each attribute depend on the interpolator implementation. See the documentation for SurfaceInput, SurfaceOutputStandard and interpolator_impl.

local_position()#
Returns:

Object-space positions XYZ raw values. Values are alpha-blended for transparent objects, if opaque_only is disabled. Alpha channel is geometry transparency. Tensor of shape (H, W, 4).

Return type:

torch.Tensor

local_position_layered() → List[Tensor]#
mso_layered() → List[Tensor]#
nvdr_antialias(rgb: Tensor)#

Applies the anti-aliasing technique in nvdiffrast to an image. This can generate visibility gradients in differentiable rendering.

Semi-transparent objects are not well-supported by this operation.

Parameters:

rgb (torch.Tensor) – Tensor of shape (H, W, 3). RGB image to be anti-aliased. Must be rendered in this session.

Returns:

Anti-aliased image.

Return type:

torch.Tensor

pbr()#

Physically-based rendering.

Returns:

Rendered HDR RGBA image in linear space. Colors from transparent objects are alpha-blended if opaque_only is disabled. Tensor of shape (H, W, 4).

Return type:

torch.Tensor

pbr_layered() → List[Tensor]#
prepare_ibl()#

Prepares the cache (prefiltered mips) for IBL.

The return value can be provided via set_prepare_ibl for other Sessions if any ImageEnvironmentLight in the scene haven’t been changed and are not requiring gradient to improve render performance.

rasterize()#

Rasterize the scene. Each layer is a tensor of shape (1, H, W, 4). xyzw means (u, v, z-depth, triangle_index).

u, v is defined as the barycentric coordinates:

\(u \cdot v_1 + v \cdot v_2 + (1 - u - v) \cdot v_3 = p\).

The layers in the front in the list are the layers in the front (closer to the camera) in 3D.

Returns:

rasterized layers. Tensors of shape (1, H, W, 4).

Return type:

List[torch.Tensor]

set_prepare_ibl(cache)#
world_position()#
Returns:

World-space positions XYZ raw values. Values are alpha-blended for transparent objects, if opaque_only is disabled. Alpha channel is geometry transparency. Tensor of shape (H, W, 4).

Return type:

torch.Tensor

world_position_layered() → List[Tensor]#
world_space_normal()#
Returns:

World-space normals. Raw values of unit vectors. Values are alpha-blended for transparent objects, if opaque_only is disabled. Alpha channel is geometry transparency. Tensor of shape (H, W, 4).

Return type:

torch.Tensor

world_space_normal_layered() → List[Tensor]#
class diffrp.rendering.surface_deferred.SurfaceDeferredRenderSessionOptions(max_layers: int = 0, interpolator_impl: Literal['full_screen', 'stencil_masked'] = 'stencil_masked', deterministic: bool = True, ibl_specular_samples: int = 512, ibl_specular_base_resolution: int = 256, ibl_specular_mip_levels: int = 5, ibl_diffuse_sample_resolution: int = 64, ibl_diffuse_base_resolution: int = 16)#

Bases: object

Options for SurfaceDeferredRenderSession.

Parameters:

  • max_layers (int) – Maximum layers to be composed for semi-transparency. 0 or negative values mean unlimited. Ignored if the render session is created in opaque_only mode (default). Otherwise, defaults to 0 (unlimited). There is a technical upper-bound of 32767 layers for composition. However, in almost all cases the layers are bounded by your GPU memory, and there will not be 32767 layers in any realistic scenes, so this value will never be reached.

  • interpolator_impl (str) –

    One of ‘stencil_masked’ and ‘full_screen’.
    Specifies the implementation of the interpolator. Defaults to ‘stencil_masked’.
    For ‘stencil_masked’ interpolator, for each material, only the active pixels go through the material call. This is memory and compute efficient but introduces more GPU pipeline flushes between material calls. You will see a flattened batched pixels in your material shade function, which means the pixel input and output tensors will have shape (B, C), where B is number of flattened pixels and C is number of channels.
    ’full_screen’ is the legacy implementation from DiffRP 0.0.1. For each material, all pixels including pixels outside the material region that are discarded afterwards. Pixel input and output tensors in materials shall have shape (B, H, W, C). This implementation uses much more memory and compute, but may be slightly more efficient due to fewer GPU pipeline flushes when there is only one material.

  • deterministic (bool) – Try to make the render deterministic. Defaults to True. Usually the variance is low enough for all operations in this deferred pipeline, but in extraordinary cases if you want to accumulate results between different calls to reduce variance for sampling processes, you may turn it off.

  • ibl_specular_samples (int) – Sample count for specular lighting in PBR Image-based lighting (IBL). Defaults to 512. Higher samples can reduce bias for environment specular lighting, but takes linearly more memory.

  • ibl_specular_base_resolution (int) – Resolution for specular lighting in IBL, whose effect is mostly visible for mirrors (metallic 1, roughness 0). Defaults to 256. Memory is quadratic to this parameter.

  • ibl_specular_mip_levels (int) – Levels of mip-maps for IBL specular lighting. Defaults to 5. Can reduce bias for roughness levels if increased, but takes linearly more memory.

  • ibl_diffuse_sample_resolution (int) – Sample resolution of diffuse irradiance convolution in IBL. Defaults to 64. Higher resolution can make the diffuse lighting more accurate, but this is usually not necessary. Memory is quadratic to this parameter.

  • ibl_diffuse_base_resolution (int) – Resolution of diffuse irradiance convolution computation in IBL. Defaults to 16. Higher resolution can make the diffuse lighting more accurate, but this is usually not necessary. Memory is quadratic to this parameter.

__init__(max_layers: int = 0, interpolator_impl: Literal['full_screen', 'stencil_masked'] = 'stencil_masked', deterministic: bool = True, ibl_specular_samples: int = 512, ibl_specular_base_resolution: int = 256, ibl_specular_mip_levels: int = 5, ibl_diffuse_sample_resolution: int = 64, ibl_diffuse_base_resolution: int = 16) → None#
deterministic: bool = True#
ibl_diffuse_base_resolution: int = 16#
ibl_diffuse_sample_resolution: int = 64#
ibl_specular_base_resolution: int = 256#
ibl_specular_mip_levels: int = 5#
ibl_specular_samples: int = 512#
property intepolator_impl#
interpolator_impl: Literal['full_screen', 'stencil_masked'] = 'stencil_masked'#
max_layers: int = 0#