Motionworks first developed Set Dynamics as a prototype in 2021 to measure the reach and frequency of roadside advertising assets (e.g., billboards). Since this initial prototype was developed, Motionworks has made several substantive updates to its placed-based and path-based methodologies. Motionworks implemented these updates to improve the integrity of its products and address the ongoing evolution of data privacy in the location-based services industry.

Version History

Major Methodology Updates

Deterministic vs. Stochastic Approach

Prototype Version Challenge: Monte Carlo simulations produced variable results across runs, making campaign planning difficult due to inconsistent reach and frequency estimates.

Version 1 Solution: Deterministic calculations provide reproducible results by treating each exposure as a probability value (0-1) rather than a binary outcome. Probabilities are aggregated across all exposures and binned into frequency classes.

Impact: Eliminates run-to-run variability while maintaining statistical accuracy. Average frequency correlations show R² > 0.95 between Prototype Version and Version 1 methods.

Panel Selection Enhancement

Prototype Version Limitation: Required 4 consecutive weeks of device activity, limiting panel size and geographic representation.

Version 1 Improvement: Allows devices with 1-4 weeks of activity, applying proportional weighting based on activity duration. Aligns with Placecast 2.2 methodology for consistency across Motionworks products.

Impact: Increased panel representation while maintaining data quality through weighted corrections.

Routing and Deduplication

Prototype Version Process: 5-minute deduplication prevented multiple spot exposures within viewsheds, potentially underestimating reach in high-density advertising areas.

Version 1 Enhancement: Removed arbitrary deduplication rules, allowing realistic multiple exposures along travel routes. Updated to HERE 2024 roadway network for improved accuracy.

Impact: More realistic frequency distributions, particularly for scenarios with clustered advertising inventory.

Visibility Adjustments

Prototype Version Data: Used 2021 Geopath VAI (Visibility Adjustment Index) calculations for impression probability estimates.

Version 1 Updates: Incorporates current Geopath VAI factors, generally showing better visibility adjustments compared to Prototype Version values.

Impact: More conservative impression estimates aligned with current industry standards compared to Prototype Version values.

Testing and Validation Framework

Version 1 validation employed a comprehensive systematic testing approach comparing performance against the Prototype Version methodology using 661 sample scenarios in Atlanta DMA.

Systematic Validation Approach

The validation used an incremental testing methodology where each test progressively incorporated additional Version 1 components. This controlled approach allowed researchers to isolate the impact of specific methodology changes and validate that improvements performed as intended.

Core Testing Principles:

• Incremental Implementation: Each test added one or more new methodology components while keeping others constant
• Controlled Comparison: All tests used identical input data and scenarios to ensure fair comparison
• Isolated Impact Analysis: Changes were introduced systematically to identify which components drove specific performance improvements
• Comprehensive Coverage: Testing spanned methodology, data sources, geographic scope, and temporal periods

Test Categories:

Deterministic Methodology Tests: Validated the core mathematical approach, comparing probability-based calculations against simulation methods. These tests confirmed elimination of run-to-run variability while maintaining statistical accuracy.

Data Enhancement Tests: Evaluated updated visibility factors, routing algorithms, and panel selection criteria. Results demonstrated improved measurement precision and geographic coverage.

Scalability Tests: Assessed methodology performance across different geographic scopes and time periods, from regional campaigns to national analysis and multi-year consistency.

Integration Tests: Validated the complete Version 1 methodology under various conditions, ensuring all components work together effectively for comprehensive out-of-home measurement.

Key Validation Results

Deterministic Methodology Tests

• Correlation: R² > 0.95 between Prototype Version and deterministic methods
• Frequency Impact: Deterministic approach shows higher average frequency due to inclusion of fractional exposures (0.5-1.0 range)
• Reach Consistency: Reach percentages remain stable across methods when using 0.5+ frequency threshold
• Reproducibility: Elimination of run-to-run variability while maintaining statistical accuracy

Data Enhancement Tests

Updated Visibility Factors:

• Trend: Better understanding of VAI values from Prototype Version to Version 1
• Correlation: Strong relationship maintained despite changes in absolute values
• Impact: More conservative impression estimates aligned with current Geopath standards

Enhanced Panel Selection:

• Coverage: Increased device representation through flexible activity periods
• Quality: Maintained data integrity through proportional weighting

Routing Algorithm Improvements:

• Frequency Distribution: More realistic exposure patterns in high-density advertising areas
• Geographic Accuracy: Improved path matching with HERE 2024 network
• Multiple Exposures: Better capture of successive advertising interactions along travel routes

Scalability Tests

• Geographic Expansion: Consistent methodology performance from regional to national campaigns
• Temporal Flexibility: Stable results across 7, 14, 21, and 28-day analysis periods
• Multi-Year Consistency: Year-over-year reach patterns maintained across 2022-2024
• Seasonal Validation: Q4 analysis showed consistent methodology performance

Integration Tests

• Component Interaction: All methodology improvements work effectively together
• End-to-End Validation: Complete Version 1 workflow produces reliable, industry-standard results
• Cross-Product Consistency: Alignment with Placecast 2.2 and other Motionworks methodologies

Quality Assurance Metrics

Replication Success: Prototype Version methodology replication achieved:

• 99.8% excellent quality scores (≤5% variance) for impression and frequency metrics
• 91.5% excellent quality scores for reach metrics
• 6.1% good quality scores (≤15% variance) for reach metrics

Quality Scoring:

• Excellent: <5% difference between methods
• Good: 5-15% difference
• Fair: 15-35% difference
• Poor: 35-82% difference

Performance Characteristics

Reach vs. Frequency Trends

• Short Periods (7-14 days): Version 1 method shows higher reach due to fractional exposure inclusion
• Long Periods (21-28 days): Version 1 method prevents randomness present in prototype version, leading to more balanced frequency over longer periods
• Frequency Distribution: Version 1 captures more nuanced exposure patterns through probability calculations

Implementation Impact

The evolution to Version 1 provides:

• Reproducibility: Consistent results across analysis runs
• Scalability: National and multi-market analysis capabilities
• Accuracy: Improved routing and panel methodology
• Industry Alignment: Current Geopath standards and Motionworks product consistency

This evolution maintains the core value proposition of reach and frequency analysis while providing more reliable, comprehensive, and industry-standard results for out-of-home campaign planning.