Skip to main content

Scheduling

Sundial provides flexible scheduling capabilities to keep your data fresh and up-to-date. You can configure data refreshes to run at regular intervals or trigger them based on specific events.

Scheduling Options​

Time-Based Scheduling​

Configure data refreshes to run on a schedule using:

  • Cron Expressions: Standard cron syntax for precise scheduling
  • Fixed Intervals: Simple recurring schedules (daily, weekly, hourly)

Example cron schedules:

  • 0 0 * * * - Run daily at midnight
  • 0 */4 * * * - Run every 4 hours
  • 0 8-17 * * MON-FRI - Run hourly from 8 AM to 5 PM on weekdays
  • */15 * * * * - Run every 15 minutes
  • 0 0 1 * * - Run monthly on the 1st day
  • 0 12 * * SUN - Run weekly on Sundays at noon

You can also use fixed intervals like:

  • Every 24 hours
  • Every Monday at 9am
  • First day of each month
  • Every 30 minutes during business hours

The scheduler supports both UTC and local timezones. You can specify the timezone when setting up the schedule to ensure refreshes happen at the right local time.

Best practices for scheduling:

  • Consider data freshness requirements
  • Avoid scheduling during peak usage times
  • Space out refreshes to manage system load
  • Set appropriate retry policies
  • Monitor schedule execution and failures

Event-Based Triggers​

In addition to time-based scheduling, Sundial supports event-driven data refreshes triggered by:

  • Data Events:

    • Source table updates
    • New data arrival in upstream systems
    • Schema changes
    • Data quality thresholds

  • System Events:

    • Pipeline completion
    • Resource availability
    • Configuration changes

  • External Events:

    • API webhooks
    • Message queue events
    • Custom application events

Example trigger configurations:

  • Refresh when upstream table is updated
  • Run after dependent pipeline completes
  • Trigger on new S3 file arrival
  • Execute when data quality check passes
  • Start on message queue notification

Best practices for event triggers:

  • Define clear event conditions
  • Handle duplicate/missed events
  • Set appropriate timeouts
  • Monitor trigger latency
  • Implement error handling
  • Test trigger reliability

You can combine time and event-based scheduling to create hybrid refresh strategies that balance data freshness with system resources.