Lightplane Renderer API

The renderer API consists of two main components:

LightplaneRenderer module which holds learnable parameters of the mlp decoder and can be reused to render different input grids
lightplane_renderer function providing the lowest-level interface to the renderer.

Visit the Lightplane Renderer section for a more detailed overview of the renderer’s functionality.

class lightplane.LightplaneRenderer(num_samples: int, color_chn: int, grid_chn: int, mlp_hidden_chn: int, mlp_n_layers_opacity: int = 2, mlp_n_layers_trunk: int = 2, mlp_n_layers_color: int = 2, use_separate_color_grid: bool = False, opacity_init_bias: float = -5.0, gain: float = 1.0, bg_color: tuple[float, ...] | float = 0.0, enable_direction_dependent_colors: bool = True, ray_embedding_num_harmonics: int | None = 3, num_samples_inf: int = 0, mask_out_of_bounds_samples: bool = False, contract_coords: bool = False, disparity_at_inf: float = 1e-05, inject_noise_sigma: float = 0.0, inject_noise_seed: int | None = None, rays_jitter_near_far: bool = False, return_log_transmittance: bool = False, triton_block_size: int = 16, triton_num_warps: int = 4, use_naive_impl: bool = False)[source]

__init__(num_samples: int, color_chn: int, grid_chn: int, mlp_hidden_chn: int, mlp_n_layers_opacity: int = 2, mlp_n_layers_trunk: int = 2, mlp_n_layers_color: int = 2, use_separate_color_grid: bool = False, opacity_init_bias: float = -5.0, gain: float = 1.0, bg_color: tuple[float, ...] | float = 0.0, enable_direction_dependent_colors: bool = True, ray_embedding_num_harmonics: int | None = 3, num_samples_inf: int = 0, mask_out_of_bounds_samples: bool = False, contract_coords: bool = False, disparity_at_inf: float = 1e-05, inject_noise_sigma: float = 0.0, inject_noise_seed: int | None = None, rays_jitter_near_far: bool = False, return_log_transmittance: bool = False, triton_block_size: int = 16, triton_num_warps: int = 4, use_naive_impl: bool = False) → None[source]

This is the Pytorch Module for the Lightplane Renderer. It uses lightplane_renderer as the core function for rendering and automatically initialize the parameters for the MLPs used in the renderer.

Parameters:

num_samples – Number of samples to render.
color_chn – Number of channels for the rendererd color.
grid_chn – Number of channels for the 3D grid to be rendered.
mlp_hidden_chn – Number of hidden channels for all MLPs.
mlp_n_layers_opacity – Number of layers for opacity_mlp.
mlp_n_layers_trunk – Number of layers for trunk_mlp.
mlp_n_layers_color – Number of layers for color_mlp.
use_separate_color_grid – Whether using a separate grid-list for colors.
opacity_init_bias – Initial bias for opacity_mlp.
gain – gain for the lightplane_renderer.
bg_color – Background color.
enable_direction_dependent_colors – Enable ray-direction dependent rendering.
ray_embedding_num_harmonics – Level of harmonic functions for ray-direction embedding. Setting ray_embedding_num_harmonics=0 will only use the ray direction as the ray encoding. A value bigger than 0 will append the harmonic embedding to the ray direction. Finally, setting ray_embedding_num_harmonics=None will use the ray encoding field from the Rays object input to forward.
num_samples_inf – Number of samples beyond the far plane.
mask_out_of_bounds_samples – Whether to mask out-of-bounds samples.
contract_coords – Whether to contract the coordinates as in MeRF.
disparity_at_inf – The beyond-far samples (their number-per-ray is determined by num_samples_inf) are sampled in the disparity space in range[far, 1 / disparity_at_inf].
inject_noise_sigma – Standard deviation of the opacity noise to inject.
inject_noise_seed – Seed for the opacity noise to inject.
rays_jitter_near_far – Whether to jitter the near and far planes uniformly in range [-delta, delta].
return_log_transmittance – Whether to return the log transmittance instead of the [0, 1] alpha mask.
use_naive_impl – Whether to use the naive pytorch implementation instead of triton.
triton_block_size – Block size for triton.
triton_num_warps – Number of warps for triton.

eval_decoder_at_points(pts: Tensor, pts_to_grid_idx: Tensor, rays_encoding: Tensor | None, feature_grid: tuple[Tensor, ...], color_feature_grid: tuple[Tensor, ...] | None = None, scaffold: Tensor | None = None, gain: float | None = None, mask_out_of_bounds_samples: bool | None = None, contract_coords: bool | None = None, directions: Tensor | None = None) → tuple[Tensor, Tensor][source]

Evaluates the MLP at the given points.

Parameters:

pts – Points to evaluate the MLP at, shape [n_rays, n_pts, 3].
pts_to_grid_idx – The grid index of each point, long tensor of shape [n_pts, ].
rays_encoding – Ray encoding for each point.
feature_grid – feature grid.
color_feature_grid – color feature grid.
scaffold – scaffold for the MLP.
gain – gain for the lightplane_renderer.
mask_out_of_bounds_samples – Whether to mask out-of-bounds samples.
contract_coords – Whether to contract the coordinates as in MeRF.
directions – Per-ray viewing directions [n_rays, 3]; can be passed instead of rays_encoding.

get_decoder_params() → DecoderParams[source]: Helper function to get the DecoderParams object from the module.

get_decoder_params_list() → Tuple[Tuple[Tensor, ...], Tuple[Tensor, ...], Tuple[Tensor, ...], Tuple[Tensor, ...], Tuple[Tensor, ...], Tuple[Tensor, ...]][source]

Helper function to get the list of weight matrices and bias vectors of the MLPs in the renderer’s decoder:

Returns:

weights_trunk – Weight matrices of the trunk MLP.
biases_trunk – Bias vectors of the trunk MLP.
weights_opacity – Weight matrices of the opacity MLP.
biases_opacity – Bias vectors of the opacity MLP.
weights_color – Weight matrices of the color MLP.
biases_color – Bias vectors of the color MLP.

eval_opacity_at_points(pts: Tensor, pts_to_grid_idx: Tensor, feature_grid: tuple[Tensor, ...] | Tensor, scaffold: Tensor | None = None, gain: float | None = None, mask_out_of_bounds_samples: bool | None = None, grid_sizes: list[list[int]] | None = None)[source]

Calcualte the opacities at the given points.

Parameters:

pts – Points to evaluate at, shape [n_rays, n_pts, 3].
pts_to_grid_idx – The grid index of each point, long tensor of shape [n_pts, ].
feature_grid – Feature grid to render from.
scaffold – scaffold for the MLP.
gain – gain for the lightplane_renderer.
mask_out_of_bounds_samples – Whether to mask out-of-bounds samples.
grid_sizes – The size of the feature_grid. Only required if feature_grid is a 2D tensor.

Returns:

results – Opacities at the given points. [n_rays, n_pts].

calculate_scaffold(feature_grid: tuple[Tensor, ...] | Tensor, scaffold_size: tuple[int, int, int, int], device, threshold: float = 1e-07, grid_sizes: list[tuple[int, int, int, int, int]] | None = None, dilate_scaffold: int = 2)[source]

Calculate scaffold by sampling voxel grid with the shape of scaffold_size and prunning points whose opacities are below threshold.

Parameters:

feature_grid – Feature grid to render from.
scaffold_size –
The shape of scaffold. scaffold should be a voxel grid with the same batch size as feature_grid. Example:
```
scaffold_size = [B, D, H, W]
```
device – Tensor device.
threshold – The opacity threshold to prune points.
grid_sizes – The size of the feature_grid. Only required if feature_grid is a 2D tensor.
dilate_scaffold – Dilate opacities before thresholding to increase the extent of the occupied regions in the scaffold.

Execute rendering with Lightplane Renderer.

Parameters:

rays – Rays to render.
feature_grid – Feature grid to render from.
color_feature_grid – Color feature grid. If provided, the color will be rendered based on this grid.
scaffold – scaffold for the MLP.
grid_sizes – The size of the feature_grid. Only required if feature_grid is a 2D tensor.
color_grid_sizes – The size of the color_feature_grid. Only required if color_feature_grid is a 2D tensor.
bg_color – Background color.
num_samples – Number of samples to render.
gain – gain for the ‘lightplane_renderer’.
num_samples_inf – Number of samples beyond the far plane.
mask_out_of_bounds_samples – Whether to mask out-of-bounds samples.
contract_coords – Whether to contract the coordinates as in MeRF.
disparity_at_inf – The beyond-far samples (their number-per-ray is determined by num_samples_inf) are sampled in the disparity space in range[far, 1 / disparity_at_inf].
inject_noise_sigma – Standard deviation of the opacity noise to inject.
inject_noise_seed – Seed for the opacity noise to inject.
rays_jitter_near_far – Whether to jitter the near and far planes uniformly in range [0, delta].
return_log_transmittance – Whether to return the log transmittance instead of the [0, 1] alpha mask.
regenerate_code – Whether to regenerate the code for the lightplane.

Returns:

ray_length_render – The rendered ray length.
alpha – The alpha mask, either in range [0, 1] if
`return_log_transmittance=False` else real-valued log transmittance.
feature_render – The rendered feature.

lightplane.lightplane_renderer(rays: Rays, grid: tuple[Tensor, ...] | Tensor, decoder_params: DecoderParams, num_samples: int, gain: float, num_samples_inf: int = 0, mask_out_of_bounds_samples: bool = False, contract_coords: bool = False, disparity_at_inf: float = 1e-05, inject_noise_sigma: float = 0.0, inject_noise_seed: int | None = None, scaffold: Tensor | None = None, color_grid: tuple[Tensor, ...] | Tensor | None = None, grid_sizes: list[list[int]] | None = None, color_grid_sizes: list[list[int]] | None = None, regenerate_code: bool = False, triton_block_size: int = 16, triton_num_warps: int = 4) → tuple[Tensor, Tensor, Tensor][source]

This is the main functional interface for the Lightplane Renderer. It outputs the the final color c, negative_log_transmittance T_N and the expected ray-termination length r (analogous to depth) of each ray’s pixel.

For N=num_samples equispaced 3D points pt_3d_i between the near and far ray-lengths, it samples the feature f_i(x_i) of 3D point pt_3d_i =x_i from the grid-list grid and calculate its corresponding renderering results.

There are three MLPs: trunk_mlp, color_mlp and opacity_mlp, whose parameters are specified in decoder_params.

trunk_mlp: Regresses the features from the f_i(x_i):
```
e_i(x_i) = trunk_mlp(f_i(x_i))
```
color_mlp: Regresses target features (e.g. final colors) from the output of trunk_mlp with ray encoding as additional input. The dimension of the output is color_chn:
```
c_i(x_i) = color_mlp(e_i(x_i) + ray_encoding)
```
opacity_mlp: Regresses the opacity scalar from the output of trunk_mlp:
```
o_i(x_i) = opacity_mlp(e_i(x_i))
```

Parameters:

rays –
The rays to render features. It is an instance of Rays, with fields directions, origins, grid_idx, near, far, and encoding. grid_idx indicates the batch index of the 3D grid to sample features from:
```
x_i = rays.origins[i] + (rays.near[i] + delta_i) * rays.direction[i]
```
grid –
Grid-list (a list of 3D grids) to sample features from. Features are sampled from each 3D grid in the set and summed up as the final feature. grid contains N tensors, each with the shape:
```
[[B, D_1, H_1, W_1, C], ... , [B, D_N, H_N, W_N, C]]
```
Each tensor must have 5 dimensions and all tensors should have the same batch size B and feature dimension C.

Example

If grid is a single Voxel grid:
```
grid = [torch.tensor([B, D, H, W, C])]
```
If grid is a triplane:
```
grid = [
    torch.tensor([B, 1, H, W, C]),
    torch.tensor([B, D, 1, W, C]),
    torch.tensor([B, D, H, 1, C]),
]
```
lightplane_renderer can also work with grid as a 2D tensor, which is a stacked tensor from the grid-list grid, with the shape [sum_(i=1..N)(B * D_i * H_i * W_i), C]. In this case, the grid_sizes must be provided to specify the shape of each grid.

Note

The 2D tensor can be obtained from lightplane.flatten_grid(grid) to flatten the list of tensors and to also obtain the grid_sizes argument.

Note

Using 2D tensor inputs improves memory-effciency when grid-list is large in memory.
decoder_params – The parameters of the decoder MLPs: trunk_mlp, color_mlp, and opacity_mlp.
num_samples –
The number of sampled points along the ray. The samples are equispaced between rays.near and rays.far. More specifically, the j-th 3d point x_ij along i-th ray is defined as follows:
```
x_ij = rays.origins[i] + (rays.near[i] + j * delta_i) * rays.direction[i],
    where:
        delta_i = (rays.far[i] - rays.near[i]) / num_samples
```
gain –
A constant to scale the transmittance T_i of i-the point along a ray:
```
T_i = exp(-gain * sum_{j=1}^{i} o(x_ij) * delta_i)
```
num_samples_inf –
The number of background samples along the ray. The first background sample is placed at rays.far, and the samples are spaced in the disparity space until reaching the disparity of disparity_at_inf.

More specifically, the j-th background 3d point b_ij along i-th ray is defined as follows:
```
b_ij = rays.origins[i] + (rays.far[i] + j * bg_delta_ij) * rays.direction[i],
    where:
        bg_delta_ij = 1 / disparity_ij
        disparity_ij = linspace(1, disparity_at_inf, num_samples_inf)[j]
```
These samples are additional to num_samples, i.e. the total number of samples along a ray is num_samples + num_samples_inf.
mask_out_of_bounds_samples – Whether to mask samples that fall outside the [-1, 1] cube (does not apply when contraction with contract_coords is enabled).

contract_coords –

Whether to map the coordinates of the rendered points to always fall into the [-1, 1] cube. The contraction is implemented as in MeRF [1]:

                x[k]                       if |x|_inf <= 1
contract(x)[k] = x[k] / |x|_inf             if x_k != |x|_inf > 1
                (2 - 1/x[k]) x_k / |x_k|   if x_k = |x|_inf > 1

Note

The contraction is useful for representing unbounded scenes. E.g. outdoor captures where the scene extends to infinity.

disparity_at_inf – The disparity value at infinity.
inject_noise_sigma – The variance of opacity noise to inject.
inject_noise_seed – The seed of the random noise to inject.
scaffold – A voxel grid with shape [B, D, H, W], indicating the occupancy of the 3D space. If provided, the renderer will only render the points that are not empty in the scaffold.
color_grid –
Another grid-list (a list of 3D grids) storing color features. If provided, the renderer will regress the color from features sampled from color_grid, using color_mlp.

Similar to grid, color_grid could also be a 2D tensor with color_grid_sizes provided. color_grid should be the same type as grid.
grid_sizes –
It specifies the size of grid. It is optional when grid is a grid-list, but required when grid is a 2D tensor. Example:
```
grid_sizes = [[B, D_1, H_1, W_1, C], ... , [B, D_N, H_N, W_N, C]].
```
color_grid_sizes –
It specifies the size of color_grid when color_grid is a 2D tensor. It is optional when color_grid is a grid-list, but required when color_grid is a 2D tensor. Example:
```
color_grid_sizes = [[B, D_1, H_1, W_1, C], ... , [B, D_N, H_N, W_N, C]]
```
regenerate_code – If True, forces the regeneration of the triton code.
triton_block_size – The block size for Triton. Has to be higher than 16.
triton_num_warps – The number of warps for Triton.

Returns:

ray_length_render – The rendered ray-termination length r (i.e. distance along the ray).
negative_log_transmittances – The negative log transmittances of the ray.
feature_render – The rendered features of the ray.

References

[1] MERF: Memory-Efficient Radiance Fields for Real-time View Synthesis in Unbounded Scenes, https://arxiv.org/abs/2302.12249