Lightplane Splatter API

The renderer API consists of two main components:

• LightplaneSplatter module which can be reused to splat features to different output grids given input rays.

• LightplaneMLPSplatter module which holds learnable parameters of the splatting mlp and which can be reused to splat features to different output grids given input rays and input grids.

• lightplane_splatter, lightplane_mlp_splatter functions providing the lowest-level interface to the splatter and the mlp-based splatter respectively.

Visit the Lightplane Splatter section for a more detailed overview of the splatter’s functionality.

class lightplane.LightplaneSplatter(num_samples: int, grid_chn: int, num_samples_inf: int = 0, mask_out_of_bounds_samples: bool = False, contract_coords: bool = False, disparity_at_inf: float = 1e-05, rays_jitter_near_far: bool = False, triton_block_size: int = 16, triton_num_warps: int = 4, use_naive_impl: bool = False)

__init__(num_samples: int, grid_chn: int, num_samples_inf: int = 0, mask_out_of_bounds_samples: bool = False, contract_coords: bool = False, disparity_at_inf: float = 1e-05, rays_jitter_near_far: bool = False, triton_block_size: int = 16, triton_num_warps: int = 4, use_naive_impl: bool = False)

This is the Pytorch Module for the Lightplane Splatter. It uses lightplane_splatter as the core function and directly splats rays.encoding to a zero-initialized output grid-list, output_grid.

Parameters:

• num_samples – Number of samples to splat.

• grid_chn – Number of channels in the output_grid.

• num_samples_inf – Number of samples beyond the far plane.

• mask_out_of_bounds_samples – Whether to mask out-of-bounds samples.

• contract_coords – Whether to contract the coordinates as in MeRF

• disparity_at_inf – The beyond-far samples (their number-per-ray is determined by num_samples_inf) are sampled in the disparity space in range[far, 1 / disparity_at_inf].

• rays_jitter_near_far – Whether to jitter the near and far planes uniformly in range [-delta, delta].

• triton_block_size – Block size for triton.

• triton_num_warps – Number of warps for triton.

• use_naive_impl – Whether to use the naive pytorch implementation.

get_splatter_params() → SplatterParams

Helper function to get the splatter parameters.

forward(rays: Rays, grid_size: list[tuple[int, int, int, int, int]], num_samples: int | None = None, num_samples_inf: int | None = None, mask_out_of_bounds_samples: bool | None = None, contract_coords: bool | None = None, disparity_at_inf: float | None = None, rays_jitter_near_far: bool | None = None, return_list: bool = True, regenerate_code: bool = False)

Forward function for splatting rays into a ‘output_grid’.

Parameters:

• rays – Rays to splat. rays.encoding is splatted to the output_grid.

• grid_size – List of tuples specifying the grid sizes of output_grid.

• num_samples – Number of samples for splatting.

• num_samples_inf – Number of samples beyond the far plane.

• mask_out_of_bounds_samples – Whether to mask out-of-bounds samples.

• contract_coords – Whether to contract the coordinates as in MeRF.

• disparity_at_inf – The beyond-far samples (their number-per-ray is determined by num_samples_inf) are sampled in the disparity space in range[far, 1 / disparity_at_inf].

• rays_jitter_near_far – Whether to jitter the near and far planes uniformly in range [-delta, delta].

• return_list – Whether to return a list of grids or a stacked tensor.

• regenerate_code – Whether to regenerate the code for the splatter.

Returns:

output_grid – splatted grid.

class lightplane.LightplaneMLPSplatter(num_samples: int, grid_chn: int, input_grid_chn: int = 32, mlp_hidden_chn: int = 32, mlp_n_layers: int = 2, num_samples_inf: int = 0, mask_out_of_bounds_samples: bool = False, contract_coords: bool = False, disparity_at_inf: float = 1e-05, rays_jitter_near_far: bool = False, triton_block_size: int = 16, triton_num_warps: int = 4, use_naive_impl: bool = False)

__init__(num_samples: int, grid_chn: int, input_grid_chn: int = 32, mlp_hidden_chn: int = 32, mlp_n_layers: int = 2, num_samples_inf: int = 0, mask_out_of_bounds_samples: bool = False, contract_coords: bool = False, disparity_at_inf: float = 1e-05, rays_jitter_near_far: bool = False, triton_block_size: int = 16, triton_num_warps: int = 4, use_naive_impl: bool = False)

This is the Pytorch Module for the Lightplane Splatter. It uses lightplane_mlp_splatter as the core function and samples the point feature from the corresponding prior input grid input_grid, adds the sampled feature to the encoding of the ray, passes the latter through an MLP, and splats the MLP output to the grid-list output_grid.

Parameters:

• num_samples – Number of samples to splat.

• grid_chn – Number of channels in the output_grid.

• input_grid_chn – Number of channels in the input_grid. It should be the same as the number of channels for rays.encoding.

• mlp_hidden_chn – Number of hidden channels in the MLP.

• mlp_n_layers – Number of layers in the MLP.

• num_samples_inf – Number of samples beyond the far plane.

• mask_out_of_bounds_samples – Whether to mask out-of-bounds samples.

• contract_coords – Whether to contract the coordinates as in MeRF

• disparity_at_inf – The beyond-far samples (their number-per-ray is determined by num_samples_inf) are sampled in the disparity space in range[far, 1 / disparity_at_inf].

• rays_jitter_near_far – Whether to jitter the near and far planes uniformly in range [-delta, delta].

• triton_block_size – Block size for triton.

• triton_num_warps – Number of warps for triton.

• use_naive_impl – Whether to use the naive pytorch implementation.

get_splatter_params() → SplatterParams

Helper function to get the splatter parameters.

forward(rays: Rays, grid_size: list[tuple[int, int, int, int, int]], input_grid: tuple[Tensor, ...] | Tensor, num_samples: int | None = None, num_samples_inf: int | None = None, mask_out_of_bounds_samples: bool | None = None, contract_coords: bool | None = None, disparity_at_inf: float | None = None, input_grid_sizes: list[list[int]] | None = None, rays_jitter_near_far: bool | None = None, return_list: bool = True, regenerate_code: bool = False)

Forward function for splatting rays into a ‘output_grid’ with an MLP and input_grid as the prior grid.

Parameters:

• rays – Rays to splat. rays.encoding is splatted to the output_grid.

• grid_size – List of tuples specifying the grid sizes of output_grid.

• input_grid – Grids to sample the point feature from.

• num_samples – Number of samples for splatting.

• num_samples_inf – Number of samples beyond the far plane.

• mask_out_of_bounds_samples – Whether to mask out-of-bounds samples.

• contract_coords – Whether to contract the coordinates as in MeRF.

• disparity_at_inf – The beyond-far samples (their number-per-ray is determined by num_samples_inf) are sampled in the disparity space in range[far, 1 / disparity_at_inf].

• input_grid_sizes – The size of the input_grid. Only required if input_grid is a 2D tensor.

• rays_jitter_near_far – Whether to jitter the near and far planes uniformly in range [-delta, delta].

• return_list – Whether to return a list of grids or a stacked tensor.

• regenerate_code – Whether to regenerate the code for the splatter.

Returns:

output_grid – splatted grid.

lightplane.lightplane_splatter(rays: Rays, output_grid_size: list[tuple[int, int, int, int, int]], num_samples: int, num_samples_inf: int = 0, mask_out_of_bounds_samples: bool = False, contract_coords: bool = False, disparity_at_inf: float = 1e-05, return_list: bool = True, regenerate_code: bool = False, triton_block_size: int = 16, triton_num_warps: int = 4) → Tensor | list[Tensor]

This is the functional interface for the Lightplane Splatter. For each point along the ray, it splats rays.encoding to a zero-initialized output grid-list, output_grid, where the shape of the output grid-list is specified by output_grid_size. It utilizes the ray-marched splatting, where the num_samples 3D points are equispaced between rays.near and rays.far. It is useful for directly splatting ray encodings

It follows:

ray encoding -> splat -> output_grid

Parameters:

• rays –

  The rays to splat feature. It is an instance of Rays, including directions, origins, grid_idx, near, far, and encoding. grid_idx indicates the batch index of the 3D grid to sample features from:

  x_i = rays.origins[i] + (rays.near[i] + delta_i) * rays.direction[i]
  

  endcoding is the feature to splat for each ray.

• output_grid_size –

  The sizes of the output_grid to be splatted. It is a list of tuples, where each tuple is the shape of the corresponding grid,in the form (B, D, H, W, C), where C has to be the same for all output grids. Example:

  output_grid_size = [(1, 64, 64, 64, 256), (1, 32, 1, 32, 256)]
  

• num_samples –

  The number of points to be splatted along the ray. The samples are equispaced between rays.near and rays.far. More specifically, the j-th 3d point x_ij along i-th ray is defined as follows:

  x_ij = rays.origins[i] + (rays.near[i] + j * delta_i) * rays.direction[i],
      where:
          delta_i = (rays.far[i] - rays.near[i]) / num_samples
  

• num_samples_inf –

  The number of points in the background to be splatted. The first background sample is placed at rays.far, and the samples are spaced in the disparity space until reaching the disparity of disparity_at_inf. More specifically, the j-th background 3d point b_ij along i-th ray is defined as follows:

  b_ij = rays.origins[i] + (rays.far[i] + j * bg_delta_ij) * rays.direction[i],
      where:
          bg_delta_ij = 1 / disparity_ij
          disparity_ij = linspace(1, disparity_at_inf, num_samples_inf)[j]
  

  These samples are additional to num_samples, i.e. the total number of samples along a ray is num_samples + num_samples_inf.

• mask_out_of_bounds_samples – Whether to mask samples that fall outside the [-1, 1] cube (does not apply when contraction with contract_coords is enabled).

• contract_coords –

  Whether to map the coordinates of the splatted points to always fall into the [-1, 1] cube. The contraction is implemented as in MeRF [1]:

                  x[k]                       if |x|_inf <= 1
  contract(x)[k] = x[k] / |x|_inf             if x_k != |x|_inf > 1
                  (2 - 1/x[k]) x_k / |x_k|   if x_k = |x|_inf > 1
  

  Note: The contraction is useful for representing unbounded scenes. E.g. outdoor captures where the scene extends to infinity.

• disparity_at_inf – The disparity value at infinity.

• return_list – Whether to return a list of grids containing the result of the splatting, or a tensor of stacked features. Note: Stacked features can be converted to a list of grids with the lightplane.misc_utils.unflatten_grid function.

• regenerate_code – If True, forces the regeneration of the triton code.

• triton_block_size – The block size for Triton. Has to be higher than 16.

• triton_num_warps – The number of warps for Triton.

Returns:

Union[torch.Tensor, List[torch.Tensor]] – The splatted results.

References

[1] MERF: Memory-Efficient Radiance Fields for Real-time View Synthesis in Unbounded Scenes, https://arxiv.org/abs/2302.12249

lightplane.lightplane_mlp_splatter(rays: Rays, output_grid_size: list[tuple[int, int, int, int, int]], mlp_params: SplatterParams, input_grid: tuple[Tensor, ...] | Tensor, num_samples: int, num_samples_inf: int = 0, mask_out_of_bounds_samples: bool = False, contract_coords: bool = False, disparity_at_inf: float = 1e-05, input_grid_sizes: list[list[int]] | None = None, return_list: bool = True, regenerate_code: bool = False, triton_block_size: int = 16, triton_num_warps: int = 4) → Tensor | list[Tensor]

This is the functional interface for the Lightplane Splatter. For each point along the ray, it first samples the point feature from the corresponding prior input grid input_grid, adds the sampled feature to the encoding of the ray, passes the latter through an MLP, and splats the MLP output to the grid-list output_grid. It utilizes the ray-marched splatting, where the num_samples 3D points are equispaced between rays.near and rays.far.

It follows:

input_grid -> f_i -> f_i + ray encoding -> mlp -> sf_i ->
splat -> output_grid

Parameters:

• rays –

  The rays to splat feature. It is an instance of Rays, including directions, origins, grid_idx, near, far, and encoding. grid_idx indicates the batch index of the 3D grid to sample features from:

  x_i = rays.origins[i] + (rays.near[i] + delta_i) * rays.direction[i]
  

  endcoding is the feature to splat for each ray.

• output_grid_size –

  The sizes of the output_grid to be splatted. It is a list of tuples, where each tuple is the shape of the corresponding grid,in the form (B, D, H, W, C), where C has to be the same for all output grids. Example:

  output_grid_size = [(1, 64, 64, 64, 256), (1, 32, 1, 32, 256)]
  

• mlp_params – The parameters of the MLP that processes the features sampled from input_grid.

• input_grid –

  The grid-list (a list of 3D grids) from which to sample features. Sames as grid in lightplane_renderer, it contains N grids, where each grid is a tensor of shape (B, D_i, H_i, W_i, C). input_grid should have the same feature dimension (C) as rays.encoding as they are summed together before passing through the MLP.

  Similar to grid in lightplane_renderer, input_grid could be a 2D tensor (the stacked version of the grid-list). In this case, input_grid_sizes must be provided to specify the shape of input_grid.

• num_samples –

  The number of points to be splatted along the ray. The samples are equispaced between rays.near and rays.far. More specifically, the j-th 3d point x_ij along i-th ray is defined as follows:

  x_ij = rays.origins[i] + (rays.near[i] + j * delta_i) * rays.direction[i],
      where:
          delta_i = (rays.far[i] - rays.near[i]) / num_samples
  

• num_samples_inf –

  The number of points in the background to be splatted. The first background sample is placed at rays.far, and the samples are spaced in the disparity space until reaching the disparity of disparity_at_inf. More specifically, the j-th background 3d point b_ij along i-th ray is defined as follows:

  b_ij = rays.origins[i] + (rays.far[i] + j * bg_delta_ij) * rays.direction[i],
      where:
          bg_delta_ij = 1 / disparity_ij
          disparity_ij = linspace(1, disparity_at_inf, num_samples_inf)[j]
  

  These samples are additional to num_samples, i.e. the total number of samples along a ray is num_samples + num_samples_inf.

• mask_out_of_bounds_samples – Whether to mask samples that fall outside the [-1, 1] cube (does not apply when contraction with contract_coords is enabled).

• contract_coords –

  Whether to map the coordinates of the splatted points to always fall into the [-1, 1] cube. The contraction is implemented as in MeRF [1]:

                  x[k]                       if |x|_inf <= 1
  contract(x)[k] = x[k] / |x|_inf             if x_k != |x|_inf > 1
                  (2 - 1/x[k]) x_k / |x_k|   if x_k = |x|_inf > 1
  

  Note: The contraction is useful for representing unbounded scenes. E.g. outdoor captures where the scene extends to infinity.

• disparity_at_inf – The disparity value at infinity.

• input_grid_sizes –

  It specifies the size of input_grid. It is optional when input_grid is a grid-list, but required when `input_grid`is a 2D tensor. Example:

  input_grid_sizes = [[B, D_1, H_1, W_1, C], ... , [B, D_N, H_N, W_N, C]]
  

• return_list – Whether to return a list of grids containing the result of the splatting, or a tensor of stacked features. Note: Stacked features can be converted to a list of grids with the lightplane.misc_utils.unflatten_grid function.

• regenerate_code – If True, forces the regeneration of the triton code.

• triton_block_size – The block size for Triton. Has to be higher than 16.

• triton_num_warps – The number of warps for Triton.

Returns:

Union[torch.Tensor, List[torch.Tensor]] – The splatted results.

References

[1] MERF: Memory-Efficient Radiance Fields for Real-time View Synthesis in Unbounded Scenes, https://arxiv.org/abs/2302.12249