1
u/thefoojoo2 1d ago
Most of this applies to AWS Athena as well. Serverless compute on S3 data, catalogued in Glue.
1
u/eternal_log_pose 1d ago
Yeah the adoption more or less depends on the organisation ecosystem as well. The orgs already on GCP and have analytical requirements on exabyte scale data go with BigQuery as they get committed discounts along with high level deals with google. One thing to note as well, BigQuery's main edge is managing the data in its capacitor format and querying Object storage might not be its prime feature so. The Capacitor format is highly optimized for columnar design. While Athena gives strong querying capability over object storage, lot depends on how the underlying data is structured.
1
u/AdamDobrawy 19h ago
The post's first point suggests that BigQuery is special in terms of data warehousing through separate compute and storage, but this is the standard architecture for modern cloud data warehouses. Available in BigQuery (2010) / Dreemel (2008), Snowflake (2012), and even Redshift's catch-up in Redshift RA3 (2019). Using column data format is also standard in these warehouses.
What infrastructure do you need to provision at Snowflake? It uses virtual warehouses, and you have predictable costs.
2
u/mad-data 1d ago
Note some of the information here is outdated. E.g. "tree of nodes" described in original Dremel VLDB 2010 paper changed into DAG as described in their 2020 paper.