Skip to main content

Materialization

Table materialization helps improve query performance by pre-computing and storing tables in Sundial's S3 storage. When you materialize a table, Sundial saves the results so they can be quickly accessed later, rather than having to recalculate them each time.

Table Types​

Sundial supports three main types of tables:

  1. View Tables - These are like regular SQL views - you can join tables, filter data, transform columns, and use functions just like you would in SQL
  2. Source Tables - These represent your raw data tables that are imported from your data warehouse
  3. Function Tables - These are specialized tables that handle specific business logic and transformations

Loading Data​

When loading data into tables, you have two options:

Incremental Loading​

  • Only loads new data since the last update (based on timestamps)
  • Useful for large tables that are frequently updated
  • More efficient since it only processes new data
  • Note: Materialized upstream tables are read from storage, while non-materialized ones need a full refresh

Full Refresh​

  • Reloads all data from scratch
  • Simpler but takes longer for large tables
  • Useful for smaller tables or when you need to reprocess everything
  • Recomputes all upstream tables as well

Saving Data​

When saving table data, Sundial currently supports two methods:

  1. Full Refresh - Completely replaces the existing table with new data
  2. Delete + Insert - Removes data for a specific time range and adds new data for that range

Performance Optimization​

You can improve query performance by partitioning tables based on frequently filtered columns. For example, you might partition by date if you often query specific time ranges. These partition settings can be configured in your table settings.