Naive Implementation API

Together with highly-optimized renderer and splatter implementations, we also provide naive auto-grad implementations of the renderer and splatter.

These implementations are numerically equivalent to their optimized versions, which is used for unit-testing the correctness of both forward and backward passes of the optimized versions.

lightplane.lightplane_renderer_naive(rays: Rays, grid: tuple[Tensor, ...] | Tensor, decoder_params: DecoderParams, num_samples: int, gain: float, mask_out_of_bounds_samples: bool = False, num_samples_inf: int = 0, contract_coords: bool = False, inject_noise_sigma: float = 0.0, inject_noise_seed: int | None = None, disparity_at_inf: float = 1e-05, scaffold: Tensor | None = None, color_grid: tuple[Tensor, ...] | Tensor | None = None, grid_sizes: list[list[int]] | None = None, color_grid_sizes: list[list[int]] | None = None, triton_num_warps: int = -1, triton_block_size: int = -1, regenerate_code: bool = False, checkpointing: bool = False)[source]

This is the naive implementation of the Lightplane Renderer (lightplane_renderer), which gives the same numeric results as the Triton implementation with less memory efficiency. It is useful for debugging and understanding the Triton implementation. It outputs the the final color c, negative_log_transmittance T_N and the expected ray-termination length r (analogous to depth) of each ray’s pixel.

Its arguments are the same as the Triton implementation in lightplane_renderer. Additionally, it could work using torch.torch.utils.checkpoint by setting checkpointing=True

Parameters:

  • rays –

    The rays to render features. It is an instance of Rays, including directions, origins, grid_idx, near, far, and encoding. grid_idx indicates the batch index of the 3D grid to sample features from:

    x_i = rays.origins[i] + (rays.near[i] + delta_i) * rays.direction[i]

  • grid –

    Grid-list (a list of 3D grids) to sample features from. Features are sampled from each 3D grid in the set and summed up as the final feature. grid contains N tensors, each with the shape:

    [[B, D_1, H_1, W_1, C], ... , [B, D_N, H_N, W_N, C]]

    Each tensor must have 5 dimensions and all tensors should have the same batch size B and feature dimension C.

    Example

    If grid is a single Voxel grid:

    grid = [torch.tensor([B, D, H, W, C])]

    If grid is a triplane:

    grid = [
    torch.tensor([B, 1, H, W, C]),
    torch.tensor([B, D, 1, W, C]),
    torch.tensor([B, D, H, 1, C]),
]

    lightplane_renderer can also work with grid as a 2D tensor, which is a stacked tensor from the grid-list grid, with the shape [sum_(i=1..N)(B * D_i * H_i * W_i), C]. In this case, the grid_sizes must be provided to specify the shape of each grid.

    The 2D tensor can be obtained from lightplane.flatten_grid(grid) to flatten the list of tensors and to also obtain the grid_sizes argument. This is useful for improving memory-effciency when grid-list is giant since we internally flatten the grid-list to a 2D tensor.

  • decoder_params – The parameters of the decoder, including the MLP parameters of trunk_mlp, color_mlp, and opacity_mlp.

  • num_samples –

    The number of sampling points along the ray. The samples are equispaced between rays.near and rays.far. More specifically, the j-th 3d point x_ij along i-th ray is defined as follows:

    x_ij = rays.origins[i] + (rays.near[i] + j * delta_i) * rays.direction[i],
    where:
        delta_i = (rays.far[i] - rays.near[i]) / num_samples

  • gain –

    A constant to scale the transmittance:

    T_i = exp(-gain * sum_{j=1}^{i} o(x_ij) * delta_i)

  • num_samples_inf –

    The number of background samples along the ray. The first background sample is placed at rays.far, and the samples are spaced in the disparity space until reaching the disparity of disparity_at_inf. More specifically, the j-th background 3d point b_ij along i-th ray is defined as follows:

    b_ij = rays.origins[i] + (rays.far[i] + j * bg_delta_ij) * rays.direction[i],
    where:
        bg_delta_ij = 1 / disparity_ij
        disparity_ij = linspace(1, disparity_at_inf, num_samples_inf)[j]

    These samples are additional to num_samples, i.e. the total number of samples along a ray is num_samples + num_samples_inf.

  • mask_out_of_bounds_samples – Whether to mask samples that fall outside the [-1, 1] cube (does not apply when contraction with contract_coords is enabled).

  • contract_coords –

    Whether to map the coordinates of the splatted points to always fall into the [-1, 1] cube. The contraction is implemented as in MeRF [1]:

                    x[k]                       if |x|_inf <= 1
contract(x)[k] = x[k] / |x|_inf             if x_k != |x|_inf > 1
                (2 - 1/x[k]) x_k / |x_k|   if x_k = |x|_inf > 1
    Note: The contraction is useful for representing unbounded scenes.

    E.g. outdoor captures where the scene extends to infinity.

  • disparity_at_inf – The disparity value at infinity.

  • inject_noise_sigma – The variance of opacity noise to inject.

  • inject_noise_seed – The seed of the random noise to inject.

  • scaffold – A voxel grid with shape [B, D, H, W], indicating the occupancy of the 3D space. If provided, the renderer will only render the points that are not empty in the scaffold.

  • color_grid –

    Another grid-list (a list of 3D grids) storing color features. If provided, the renderer will regress the color from features sampled from color_grid, using color_mlp.

    Similar to grid, color_grid could also be a 2D tensor with color_grid_sizes provided. color_grid should be the same type as grid.

  • grid_sizes –

    It specifies the size of grid. It is optional when grid is a grid-list, but required when grid is a 2D tensor. Example:

    grid_sizes = [[B, D_1, H_1, W_1, C], ... , [B, D_N, H_N, W_N, C]]

  • color_grid_sizes –

    It specifies the size of color_grid when color_grid

    is a 2D tensor.

    It is optional when color_grid is a grid-list, but required when color_grid is a 2D tensor. Example:

    color_grid_sizes = [[B, D_1, H_1, W_1, C], ... , [B, D_N, H_N, W_N, C]]

  • regenerate_code – Ignored, but kept for compatibility with triton api.

  • triton_block_size – Ignored, but kept for compatibility with triton api.

  • triton_num_warps – Ignored, but kept for compatibility with triton api.

  • checkpointing – Whether or not use torch.utils.checkpoint for checkpointing.

Returns:

  • ray_length_render – The rendered ray-termination length r (i.e. distance along the ray).

  • negative_log_transmittances – The negative log transmittances of the ray.

  • feature_render – The expected features of the ray.

References

[1] MERF: Memory-Efficient Radiance Fields for Real-time View Synthesis in Unbounded Scenes, https://arxiv.org/abs/2302.12249

lightplane.lightplane_splatter_naive(rays: Rays, output_grid_size: list[tuple[int, int, int, int, int]], num_samples: int, num_samples_inf: int = 0, mask_out_of_bounds_samples: bool = False, contract_coords: bool = False, disparity_at_inf: float = 1e-05, return_list: bool = True, regenerate_code: bool = False, triton_block_size: int = 16, triton_num_warps: int = 4, checkpointing: bool = False) → Tensor | list[Tensor][source]

This is the naive implementation of the Lightplane Splatter (lightplane_splatter), which gives the same numeric results as the Triton implementation with less memory efficiency. It is useful for debugging and understanding the Triton implementation.

Its arguments are the same as the Triton implementation in lightplane_splatter. Additionally, it could work using torch.torch.utils.checkpoint by setting checkpointing=True.

Please follow the docstring of lightplane.lightplane_splatter() for description of arguments and returned variables.

Note

The following method arguments are additional to the arguments of lightplane.lightplane_splatter().

Parameters:

  • regenerate_code – Ignored, but kept for compatibility with triton api

  • triton_block_size – Ignored, but kept for compatibility with triton api

  • triton_num_warps – Ignored, but kept for compatibility with triton api

  • checkpointing – Whether or not use torch.utils.checkpoint for checkpointing.

Returns:

Union[torch.Tensor, List[torch.Tensor]] – The splatted results.

References

[1] MERF: Memory-Efficient Radiance Fields for Real-time View Synthesis in Unbounded Scenes, https://arxiv.org/abs/2302.12249

lightplane.lightplane_mlp_splatter_naive(rays: Rays, output_grid_size: list[tuple[int, int, int, int, int]], mlp_params: SplatterParams, input_grid: tuple[Tensor, ...] | Tensor, num_samples: int, num_samples_inf: int = 0, mask_out_of_bounds_samples: bool = False, contract_coords: bool = False, disparity_at_inf: float = 1e-05, input_grid_sizes: list[list[int]] | None = None, return_list: bool = True, regenerate_code: bool = False, triton_block_size: int = 16, triton_num_warps: int = 4, checkpointing: bool = False) → Tensor | list[Tensor][source]

This is the naive implementation of the Lightplane Splatter with MLP and input_grid (lightplane_mlp_splatter), which gives the same numeric results as the Triton implementation with less memory efficiency. It is useful for debugging and understanding the Triton implementation.

Its arguments are the same as the Triton implementation in lightplane_mlp_splatter. Additionally, it could work using torch.torch.utils.checkpoint by setting checkpointing=True.

Please follow the docstring of lightplane.lightplane_mlp_splatter() for description of arguments and returned variables.

Note

The following method arguments are additional to the arguments of lightplane.lightplane_mlp_splatter().

Parameters:

  • regenerate_code – Ignored, but kept for compatibility with triton api

  • triton_block_size – Ignored, but kept for compatibility with triton api

  • triton_num_warps – Ignored, but kept for compatibility with triton api

  • checkpointing – Whether or not use torch.utils.checkpoint for checkpointing.

Returns:

Union[torch.Tensor, List[torch.Tensor]] – The splatted results.

References

[1] MERF: Memory-Efficient Radiance Fields for Real-time View Synthesis in Unbounded Scenes, https://arxiv.org/abs/2302.12249