# Big data and lakehouse Kit: Apache projects for small business

October 29, 2026. [Read the article](https://jwatte.com/blog/apache-big-data-lakehouse-small-business/).

## How to use this kit

Write down one question your current report cannot answer. Give your helper a copy of the smallest relevant dataset. Compare a single-machine trial with a managed service before authorizing a cluster. No project in this group is classified for ordinary owner operation; the cards assume a developer or IT provider. We have read documentation, not installed these systems.

## Trial cards

### Spark

Fits repeatable file analysis: an HVAC contractor's return-visit costs or a machine shop's rework report. Start with the [official quickstart](https://spark.apache.org/docs/latest/quick-start.html) and [hardware guidance](https://spark.apache.org/docs/latest/hardware-provisioning.html). Local mode uses one computer; our pilot is 4 vCPU and 8 GiB, not a guaranteed minimum. Java and SQL or Python skills are needed. Back up inputs, code and credentials separately; test a rerun after upgrades. In week one, reconcile one report against a manual total and time the largest job. Ask who handles failures and which data may leave the business. EMR Serverless is the managed alternative costed below.

### Flink

Fits a justified event problem: missing bakery freezer readings or incomplete delivery scans. [First steps](https://nightlies.apache.org/flink/flink-docs-stable/docs/getting-started/local_installation/) run JobManager and TaskManager through Docker; no numeric hardware minimum is published there. Our single 8 GiB host is a pilot only. IT must understand state, checkpoints and replay. Back up state and configuration, rehearse restart, and review connector compatibility before updates. In week one replay copied events and verify duplicate handling. Ask your provider for a separate resilient topology quote. Amazon's managed Flink service is costed below.

### DataFusion

Fits a developer preparing supplier comparisons for a landscaping crew or parcel-charge exceptions for a retailer. [Install the CLI](https://datafusion.apache.org/user-guide/cli/installation.html) using Cargo or the documented macOS route; Docker needs a source build. No numeric RAM floor is specified; our 4 GiB server is a test allocation. SQL and file-format skills are required. Save queries with source files, impose a memory limit, and check results after upgrades. In week one compare output against known rows and document a repeatable command. Ask who maintains the tool when its original developer leaves. Athena is an outcome alternative, not hosted DataFusion.

## Other IT project cards

The smallest setups below are evaluation starting points, not resilience guarantees. Each card's 6 to 16 hours/month is a shared platform allowance. For every card ask: who patches it, what persists outside the container, how is recovery tested, what version combinations are supported, and what does the managed bill exclude?

**Apache Ambari.** A server, database and agents on the managed hosts are the starting layout. It changes cluster configuration, so an administrator must own upgrades. A 30 person manufacturer inheriting Hadoop might keep it. [Setup source](https://ambari.apache.org/docs/3.0.0/quick-start/installation-guide/). Allowance: 6 to 16 platform hours/month. Alternative: EMR compute pricing in the Spark comparison; migration replaces some cluster management.

**Apache Amoro (Incubating).** The Compose demonstration combines the management service with storage and a processing engine. Table optimization needs tuning. A 20 person online retailer with existing analytical tables could use it. [Setup source](https://amoro.incubator.apache.org/quick-start/). Allowance: 6 to 16 platform hours/month. Alternative: Athena at $5/TB scanned for simpler reporting, without equivalent table maintenance.

**Apache Atlas.** A local trial combines Atlas, HBase and Solr. The guide recommends at least two Solr servers and three or five ZooKeeper nodes for SolrCloud; its old dependency versions need review. A 40 person distributor might need to trace which supplier file produced a report. [Setup source](https://raw.githubusercontent.com/apache/atlas/master/docs/src/documents/Setup/InstallationInstruction.md). Allowance: 6 to 16 platform hours/month. Alternative: No verified fixed price for a directly managed Atlas service; request a quote.

**Apache Bigtop.** This builds and tests a compatible Hadoop software stack. Its repository recommends virtual machines for package tests because tests install software. A regional delivery firm maintaining an inherited cluster could justify that engineering work. [Setup source](https://github.com/apache/bigtop). Allowance: 6 to 16 platform hours/month. Alternative: EMR compute pricing in the Spark comparison; avoid maintaining a distribution.

**Apache Celeborn.** A master and workers move intermediate processing data; resilient masters need the documented HA configuration. A print fulfillment company running many Spark jobs might benefit after measuring shuffle bottlenecks. [Setup source](https://celeborn.apache.org/docs/latest/deploy/). Allowance: 6 to 16 platform hours/month. Alternative: EMR compute pricing in the Spark comparison, with workload tuning still required.

**Apache Drill.** Embedded mode is the smallest route for file queries; distributed mode adds coordination and service security. A 12 person machine shop could compare exports from old machines without replacing their software. [Setup source](https://drill.apache.org/docs/embedded-mode-prerequisites/). Allowance: 6 to 16 platform hours/month. Alternative: Athena at $5/TB scanned.

**Apache Fluo.** Fluo maintains incremental computations over Accumulo and uses ZooKeeper. Its dependency stack rules out treating one application process as the whole installation. A delivery company updating route relationships could justify it. [Setup source](https://fluo.apache.org/). Allowance: 6 to 16 platform hours/month. Alternative: Managed Flink at $0.11/KPU-hour plus orchestration and storage, after rewriting the application.

**Apache Fluo YARN.** This adds YARN deployment to Fluo, so the smallest useful setup already has Fluo, Accumulo and Hadoop services. A distributor with that existing stack might keep it; a new shop should skip it. [Setup source](https://fluo.apache.org/). Allowance: 6 to 16 platform hours/month. Alternative: EMR compute pricing in the Spark comparison does not include Fluo integration.

**Apache Fluss.** The quickstart combines Fluss services, ZooKeeper, Flink and object storage in Compose. Persistent streaming tables add recovery work. A two warehouse retailer tracking stock changes might have a use for them. [Setup source](https://fluss.apache.org/docs/quickstart/flink/). Allowance: 6 to 16 platform hours/month. Alternative: Managed Flink at $0.11/KPU-hour plus overhead; storage architecture differs.

**Apache GeaFlow (Incubating).** The guide builds Java code with Maven and Docker before running graph jobs. This needs a developer and an operator. A small parcel carrier exploring connected fraud patterns might commission a trial. [Setup source](https://geaflow.incubator.apache.org/docs/guide). Allowance: 6 to 16 platform hours/month. Alternative: No verified fixed price for a managed equivalent; obtain a scoped quote.

**Apache Gravitino.** One Java 17 server can start the catalog demonstration, but connected engines and permission mapping remain separate. A 25 person manufacturer using multiple stores might need a common inventory of tables. [Setup source](https://gravitino.apache.org/docs/1.3.1/). Allowance: 6 to 16 platform hours/month. Alternative: No verified fixed price for the same catalog service; obtain a quote.

**Apache Hadoop.** The official single-node setup demonstrates Java services on Linux and warns that production needs authentication. Distributed storage introduces replication and recovery. A farm cooperative inheriting large historical datasets might retain it. [Setup source](https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/SingleCluster.html). Allowance: 6 to 16 platform hours/month. Alternative: EMR compute pricing in the Spark comparison; storage costs are additional.

**Apache Hive.** The Docker quickstart runs HiveServer2 and metastore services; durable installations also need a database and storage. A wholesaler querying years of transactions might justify it alongside existing Hadoop. [Setup source](https://hive.apache.org/development/quickstart/). Allowance: 6 to 16 platform hours/month. Alternative: Athena at $5/TB scanned for SQL reporting.

**Apache Hudi.** The quickstart adds Hudi to Spark, with file storage and record keys to maintain. Compaction and schema changes need an operator. An online retailer merging corrected order exports could use it. [Setup source](https://hudi.apache.org/docs/quick-start-guide). Allowance: 6 to 16 platform hours/month. Alternative: Athena at $5/TB for the reading side only; writes still need a pipeline.

**Apache Impala.** The documented layout has query processes, a state service and catalog service, plus a database-backed Hive metastore. Its optimal-join guidance recommends 128 GB RAM or more; a small VPS is not that design. A regional parts distributor with existing Hadoop might justify faster interactive queries. [Setup source](https://impala.apache.org/docs/build/html/topics/impala_prereqs.html). Allowance: 6 to 16 platform hours/month. Alternative: Athena at $5/TB scanned; measure latency before choosing.

**Apache Kyuubi.** The laptop quickstart starts a server and Spark engine with a JDBC client. Shared production access adds authentication and engine isolation. A 15 person data consultancy might need that gateway. [Setup source](https://kyuubi.readthedocs.io/en/v1.12.0/quick_start/quick_start.html). Allowance: 6 to 16 platform hours/month. Alternative: EMR compute pricing in the Spark comparison; shared access features need separate design.

**Apache Linkis.** The quick deployment guide calls for at least 3 GB memory for roughly six microservices, before the engines they launch. A retailer with several analyst tools might use it to share execution services. [Setup source](https://linkis.apache.org/docs/latest/deployment/deploy-quick). Allowance: 6 to 16 platform hours/month. Alternative: EMR compute pricing in the Spark comparison; this is an execution alternative, not the same gateway.

**Apache Livy.** The small route is a Livy server connected to Spark. The current start page names Scala 2.12 Spark builds, so do not assume Spark 4 compatibility. A manufacturer with an existing notebook integration might retain it. [Setup source](https://livy.apache.org/get-started). Allowance: 6 to 16 platform hours/month. Alternative: EMR compute pricing in the Spark comparison; verify session interfaces before migration.

**Apache Ozone.** The Compose guide explicitly targets development, testing and evaluation. Production object storage needs multiple services and recovery planning. A print shop retaining large production files might consider it only with a provider. [Setup source](https://ozone.apache.org/docs/quick-start/installation/docker). Allowance: 6 to 16 platform hours/month. Alternative: B2 storage at $6.95/TB/month is a simpler storage alternative, not identical semantics.

**Apache Paimon.** The Flink quickstart uses a matching connector and a local file warehouse, with two task slots for simultaneous writing and reading. A cluster needs shared storage; the guide gives no universal RAM minimum. An ecommerce brand maintaining changing inventory tables could justify it. [Setup source](https://raw.githubusercontent.com/apache/paimon/master/docs/docs/flink/quick-start.mdx). Allowance: 6 to 16 platform hours/month. Alternative: Managed Flink at $0.11/KPU-hour plus overhead; table support needs checking.

**Apache Pig.** The guide supports local execution and Hadoop execution, but still contains old Java and Hadoop requirements. A wholesaler maintaining inherited scripts needs a compatibility review before upgrading. [Setup source](https://pig.apache.org/docs/r0.18.0/start.html). Allowance: 6 to 16 platform hours/month. Alternative: EMR compute pricing in the Spark comparison; replacing Pig scripts takes separately priced work.

**Apache Polaris.** The quickstart uses an in-memory metastore and is explicitly nonproduction. Production needs durable JDBC storage such as PostgreSQL, stable keys and access checks, plus the separate warehouse and query engine. A distributor sharing tables between engines might need it. [Setup source](https://polaris.apache.org/releases/1.8.0/configuration/configuring-polaris-for-production/). Allowance: 6 to 16 platform hours/month. Alternative: No verified fixed price for equivalent managed catalog service; obtain a quote.

**Apache Samza.** The quickstart introduces a processing job and its supporting messaging environment. Stateful deployment and replay belong with IT. A delivery firm processing vehicle events might retain it where the team already knows it. [Setup source](https://samza.apache.org/startup/quick-start/latest). Allowance: 6 to 16 platform hours/month. Alternative: Managed Flink at $0.11/KPU-hour plus overhead, after code migration.

**Apache Sdap.** The local Docker guide needs 8.5 GB disk plus multiple services; production documentation uses Kubernetes charts. A small irrigation consultancy analyzing scientific gridded data might justify specialist operation. [Setup source](https://incubator-sdap-nexus.readthedocs.io/en/latest/quickstart.html). Allowance: 6 to 16 platform hours/month. Alternative: No verified fixed price for equivalent managed scientific service; obtain a quote.

**Apache Storm.** Nimbus, worker services and ZooKeeper form the documented cluster layout. Its current guide names Java 25 and Python 3. A car wash equipment provider processing fleet telemetry might justify it. [Setup source](https://storm.apache.org/releases/current/Setting-up-a-Storm-cluster.html). Allowance: 6 to 16 platform hours/month. Alternative: Managed Flink at $0.11/KPU-hour plus overhead, after code migration.

**Apache Streampark.** The installation needs Java, an application database and Flink; Hadoop or Kubernetes depends on deployment mode. A small software supplier operating customer stream jobs could use its management interface. [Setup source](https://streampark.apache.org/docs/get-started/installation). Allowance: 6 to 16 platform hours/month. Alternative: Managed Flink at $0.11/KPU-hour plus overhead.

**Apache Uniffle.** Its deployment has coordinators, shuffle servers and an engine client. Disk pressure and compatibility need monitoring. A fulfillment company with measured Spark shuffle problems might ask its provider to test it. [Setup source](https://uniffle.apache.org/docs/intro). Allowance: 6 to 16 platform hours/month. Alternative: EMR compute pricing in the Spark comparison; benchmark before assuming savings.

## Full group table

Registry dates can lag; file dates are distribution modification dates. Incubating projects are not yet fully endorsed by the ASF; see the [incubation policy](https://incubator.apache.org/policy/incubation.html). No hardware benchmark or production installation is claimed.

| Project | What it does | Who runs it | Latest recorded release and date |
|---|---|---|---|
| [Apache Ambari](https://ambari.apache.org) <!-- project:ambari --> | Installs, configures and monitors Hadoop clusters. | Your IT department or provider | [3.0.0; 2025-04-10](https://github.com/apache/ambari/releases/tag/release-3.0.0) |
| [Apache Amoro (Incubating)](https://amoro.incubator.apache.org/) <!-- project:incubator-amoro --> | Manages analytical tables stored in open data lake formats. | Your IT department or provider | [0.8.1-incubating; 2025-09-11](https://amoro.incubator.apache.org/download/) |
| [Apache Arrow](https://arrow.apache.org) <!-- project:arrow --> | Lets data tools exchange and process column based data efficiently. | Your developer | [26.0.0; 2026-10-09](https://projects.apache.org/json/foundation/projects.json) (registry) |
| [Apache Atlas](https://atlas.apache.org) <!-- project:atlas --> | Tracks where data comes from and how it is classified across data systems. | Your IT department or provider | [2.5.0; 2026-04-30](https://downloads.apache.org/atlas/2.5.0/) (file date) |
| [Apache Auron](https://auron.apache.org) <!-- project:auron --> | Accelerates large data processing workloads with native execution code. | Your developer | [8.0.0; 2026-07-18](https://github.com/apache/auron/releases/tag/v8.0.0) |
| [Apache Beam](https://beam.apache.org) <!-- project:beam --> | Lets developers define data processing jobs for different execution systems. | Your developer | [2.77.0; 2026-10-07](https://beam.apache.org/get-started/downloads/) |
| [Apache Bigtop](https://bigtop.apache.org/) <!-- project:bigtop --> | Packages and tests components used to build large data platforms. | Your IT department or provider | [3.6.0; 2026-07-10](https://downloads.apache.org/bigtop/bigtop-3.6.0/) (file date) |
| [Apache Calcite](https://calcite.apache.org) <!-- project:calcite --> | Helps developers build database query planners and SQL interfaces. | Your developer | [1.42.0; 2026-05-31](https://calcite.apache.org/downloads/) |
| [Apache CarbonData](https://carbondata.apache.org) <!-- project:carbondata --> | Provides indexed data files for large analytical workloads. | Your developer | [2.3.2; 2026-03-25](https://downloads.apache.org/carbondata/2.3.2/) (file date) |
| [Apache Celeborn](https://celeborn.apache.org) <!-- project:celeborn --> | Provides shared temporary data storage for distributed processing jobs. | Your IT department or provider | [0.7.0; 2026-08-16](https://celeborn.apache.org/download/) |
| [Apache DataFu](https://datafu.apache.org/) <!-- project:datafu --> | Supplies reusable routines for preparing and analyzing large datasets. | Your developer | [2.1.0; 2025-04-17](https://downloads.apache.org/datafu/apache-datafu-2.1.0/) (file date) |
| [Apache DataFusion](https://datafusion.apache.org) <!-- project:datafusion --> | Provides a query engine for developers building data applications. | Your developer | [55.2.0; 2026-10-07](https://downloads.apache.org/datafusion/datafusion-55.2.0/) (file date) |
| [Apache Drill](https://drill.apache.org) <!-- project:drill --> | Queries files and databases with SQL without first loading everything into one database. | Your IT department or provider | [1.22.0; 2025-06-28](https://drill.apache.org/download/) |
| [Apache Flink](https://flink.apache.org) <!-- project:flink --> | Processes incoming events and stored datasets across servers. | Your IT department or provider | [2.3.0; 2026-06-25](https://flink.apache.org/downloads/) |
| [Apache Fluo](https://fluo.apache.org) <!-- project:accumulo-fluo --> | Updates large datasets incrementally when related data changes. | Your IT department or provider | [2.0.0; 2023-04-18](https://fluo.apache.org/releases/) |
| [Apache Fluo Recipes](https://fluo.apache.org) <!-- project:accumulo-fluo_recipes --> | Provides reusable code for developers building Fluo data processing applications. | Your developer | [1.2.0; 2018-03-06](https://projects.apache.org/json/foundation/projects.json) (registry) |
| [Apache Fluo YARN](https://fluo.apache.org) <!-- project:accumulo-fluo_yarn --> | Runs Fluo data processing applications inside Hadoop resource management. | Your IT department or provider | [1.0.0; 2018-03-06](https://fluo.apache.org/releases/) |
| [Apache Fluss](https://fluss.apache.org) <!-- project:fluss --> | Stores streaming data for continuously updated analytics. | Your IT department or provider | [1.0.0; 2026-09-22](https://github.com/apache/fluss/releases/tag/v1.0.0) |
| [Apache GeaFlow (Incubating)](https://geaflow.incubator.apache.org/) <!-- project:incubator-geaflow --> | Processes changing graph data and relationships across servers. | Your IT department or provider | [0.8.0-incubating; 2026-06-10](https://downloads.apache.org/incubator/geaflow/0.8.0-incubating/) (file date) |
| [Apache Gluten](https://gluten.apache.org) <!-- project:gluten --> | Speeds up existing data processing systems with native execution engines. | Your developer | [1.7.0; 2026-08-26](https://github.com/apache/gluten/releases/tag/v1.7.0) |
| [Apache Gravitino](https://gravitino.apache.org) <!-- project:gravitino --> | Catalogs data and its access across different storage and processing systems. | Your IT department or provider | [1.3.1; 2026-09-30](https://github.com/apache/gravitino/releases/tag/v1.3.1) |
| [Apache Hadoop](https://hadoop.apache.org/) <!-- project:hadoop --> | Stores and processes large datasets across a group of servers. | Your IT department or provider | [3.5.0; 2026-04-01](https://projects.apache.org/json/foundation/projects.json) (registry) |
| [Apache Hive](https://hive.apache.org) <!-- project:hive --> | Queries large collections of stored data using SQL. | Your IT department or provider | [4.2.1; 2026-08-24](https://hive.apache.org/general/downloads/) |
| [Apache Hudi](https://hudi.apache.org) <!-- project:hudi --> | Maintains analytical datasets as new records and corrections arrive. | Your IT department or provider | [1.1.2; 2026-10-17](https://projects.apache.org/json/foundation/projects.json) (registry) |
| [Apache Iceberg](https://iceberg.apache.org) <!-- project:iceberg --> | Defines reliable analytical tables that several data engines can share. | Your developer | [1.12.0; 2026-09-29](https://projects.apache.org/json/foundation/projects.json) (registry) |
| [Apache Impala](https://impala.apache.org) <!-- project:impala --> | Runs SQL queries over large datasets stored in data platforms. | Your IT department or provider | [4.5.3; 2026-10-05](https://projects.apache.org/json/foundation/projects.json) (registry) |
| [Apache Kyuubi](https://kyuubi.apache.org) <!-- project:kyuubi --> | Shares access to large data processing engines through SQL services. | Your IT department or provider | [1.12.0; 2026-07-28](https://github.com/apache/kyuubi/releases/tag/v1.12.0) |
| [Apache Linkis](https://linkis.apache.org/) <!-- project:linkis --> | Connects data applications to shared computation and analysis engines. | Your IT department or provider | [1.8.0; 2025-10-17](https://linkis.apache.org/download/main/) |
| [Apache Livy](https://livy.apache.org) <!-- project:livy --> | Lets applications submit and monitor Spark jobs remotely. | Your IT department or provider | [0.9.0-incubating; 2026-01-09](https://downloads.apache.org/incubator/livy/0.9.0-incubating/) (file date) |
| [Apache ORC](https://orc.apache.org) <!-- project:orc --> | Stores large analytical tables in compact files designed for fast reading. | Your developer | [1.8.1; 2022-12-02](https://projects.apache.org/json/foundation/projects.json) (registry) |
| [Apache Ozone](https://ozone.apache.org/) <!-- project:ozone --> | Stores large volumes of files and objects across servers. | Your IT department or provider | [2.2.1; 2026-08-27](https://ozone.apache.org/download/) |
| [Apache Paimon](https://paimon.apache.org) <!-- project:paimon --> | Maintains analytical tables for continuously arriving data. | Your IT department or provider | [2.1.0; 2026-10-12](https://github.com/apache/paimon/releases/tag/release-2.1.0) |
| [Apache Parquet](https://parquet.apache.org) <!-- project:parquet --> | Stores analytical data in compact column based files. | Your developer | [1.13.1; 2023-05-18](https://projects.apache.org/json/foundation/projects.json) (registry) |
| [Apache Pig](https://pig.apache.org) <!-- project:pig --> | Transforms and analyzes Hadoop data using scripts. | Your IT department or provider | [0.18.0; 2025-09-15](http://pig.apache.org/releases.html) |
| [Apache Polaris](https://polaris.apache.org) <!-- project:polaris --> | Catalogs analytical tables and controls access across data engines. | Your IT department or provider | [1.8.0; 2026-09-28](https://projects.apache.org/json/foundation/projects.json) (registry) |
| [Apache Samza](https://samza.apache.org/) <!-- project:samza --> | Processes continuous streams of events across servers. | Your IT department or provider | [1.8.0; 2023-01-17](https://samza.apache.org/) |
| [Apache Sdap](https://sdap.apache.org) <!-- project:sdap --> | Analyzes scientific data stored across distributed systems. | Your IT department or provider | [1.4.0; 2024-11-04](https://sdap.apache.org/downloads) |
| [Apache Sedona](https://sedona.apache.org) <!-- project:sedona --> | Adds large scale geographic data processing to analytical applications. | Your developer | [1.9.1; 2026-08-05](https://github.com/apache/sedona/releases/tag/sedona-1.9.1) |
| [Apache Spark](https://spark.apache.org) <!-- project:spark --> | Processes large datasets for reporting, data preparation and machine learning. | Your IT department or provider | [4.2.1; 2026-10-27](https://spark.apache.org/downloads.html) |
| [Apache Storm](https://storm.apache.org/) <!-- project:storm --> | Processes continuous event streams across a cluster of servers. | Your IT department or provider | [3.1.0; 2026-09-12](https://projects.apache.org/json/foundation/projects.json) (registry) |
| [Apache Streampark](https://streampark.apache.org) <!-- project:streampark --> | Helps teams develop, deploy and manage stream processing jobs. | Your IT department or provider | [2.1.7; 2025-11-05](https://streampark.apache.org/download/) |
| [Apache Tez](https://tez.apache.org) <!-- project:tez --> | Executes linked data processing tasks within Hadoop systems. | Your developer | [0.10.5; 2025-05-30](https://projects.apache.org/json/foundation/projects.json) (registry) |
| [Apache Toree (Incubating)](https://toree.incubator.apache.org/) <!-- project:incubator-toree --> | Connects interactive applications and notebooks to Spark processing. | Your developer | [0.5.0-incubating; 2022-02-16](https://downloads.apache.org/incubator/toree/0.5.0-incubating/toree/) (file date) |
| [Apache Uniffle](https://uniffle.apache.org) <!-- project:uniffle --> | Moves intermediate data between tasks in large processing jobs. | Your IT department or provider | [0.10.0; 2025-09-25](https://github.com/apache/uniffle/releases/tag/v0.10.0) |
| [Apache Wayang](https://wayang.apache.org) <!-- project:wayang --> | Coordinates analytical work across different processing engines. | Your developer | [1.1.1; 2026-02-11](https://downloads.apache.org/wayang/1.1.1/) (file date) |
| [Apache XTable (Incubating)](https://xtable.incubator.apache.org/) <!-- project:incubator-xtable --> | Translates analytical table metadata so different data tools can share datasets. | Your developer | [0.4.0-incubating; 2026-08-24](https://github.com/apache/incubator-xtable/releases/tag/0.4.0-incubating) |

## Cost worksheet

Setup hours: low ___, high ___. Administration hours/month: low ___, high ___. Nodes and server price: ___. Retained backup GB: low ___, high ___. Managed service units and billing term: ___. Existing common costs excluded on both sides: ___. Migration and custom development quote: ___.

Prices are US dollars before tax, read October 29, 2026. Our [labor basis](https://data.bls.gov/oesprofile/?areas=INDUSTRY,STATE,MSA&major_group=150000&measure=01&occupation=151244&year=2025) is $47.66 divided by the [70.0% wage share](https://data.bls.gov/timeseries/CMU2020000000000P), rounded to $68.09/hour. A [DigitalOcean Medium server](https://www.digitalocean.com/pricing/droplets) is $48/month for 4 vCPU, 8 GiB RAM and 160 GiB disk; Small is $24 for 2 vCPU, 4 GiB and 80 GiB. [B2 backups](https://www.backblaze.com/cloud-storage/pricing) cost max(0, retained GB minus 10) x $0.00695/month. Our low and high retained volumes are 50 and 200 GB, costing $0.28 and $1.32 before rounding totals.

For a packaged single-machine pilot, our assumptions are 4 to 12 setup hours and 1 to 3 administration hours/month. For an IT platform, they are 24 to 60 and 6 to 16. Setup = hours x $68.09. Monthly = servers + backup + administration hours x $68.09. The 36 month total = setup + 36 x monthly; savings = alternative total minus self hosted total. Break even = ceiling(setup divided by monthly savings), or never if there are no monthly savings. An outside IT provider bills more than this employee compensation proxy: get a quote.

The tables compare infrastructure operation for already written jobs. Custom code, cleanup, training, migration, existing source storage, taxes, excess transfer and incident response are excluded on both sides. Alternative setup is assumed zero at both ends because the illustration assumes an existing account and working job. That assumption favors the alternative; it is not a promise of free implementation.

### Filled Spark example

[EMR Serverless](https://aws.amazon.com/emr/pricing/): 4 aggregate vCPU and 8 GB for 20 to 80 hours; hours x (4 x $0.052624 + 8 x $0.0057785). One Medium self hosted server, 4 to 12 setup hours and 1 to 3 monthly hours. This is an existing working-job comparison, not a migration quote.

| Cost | Low labor case | High labor case |
|---|---:|---:|
| Setup | $272 | $817 |
| Monthly operations | $116 | $254 |
| Self hosted, 36 months | $4,462 | $9,946 |
| Alternative monthly | $5 | $21 |
| Alternative, 36 months | $185 | $739 |
| Savings from self hosting | Negative $4,277 | Negative $9,207 |
| Break even month | Never | Never |

### Setup time when Claude or ChatGPT does the typing

We keep the Spark report pilot: one $48 Medium server, 50 to 200 GB of backups and already written jobs. Our assumed setup shares are 50% and 80% of manual hours; monthly shares are 75% and 100%. These single-machine assumptions retain IT oversight. Included human review is 1 to 3 setup hours and 0.5 to 1.5 monthly hours. Prompting, corrections and testing count; unattended agent time does not. These are planning estimates, not measured Apache installations.

| Spark pilot measure | Low case | High case |
|---|---:|---:|
| Manual setup hours | 4 | 12 |
| AI assisted setup hours | 2 | 9.6 |
| Manual administration hours/month | 1 | 3 |
| AI assisted administration hours/month | 0.75 | 3 |
| Manual, 36 months | $4,462 | $9,946 |
| Claude Pro subscription, 36 months | $3,913 | $9,983 |
| ChatGPT Plus subscription, 36 months | $3,913 | $9,983 |
| API: Claude Opus 5.5, 36 months | $3,727 | $9,897 |
| API: GPT-6 Astra, 36 months | $3,750 | $10,068 |


Manual total = S x L + 36 x (H + A x L). AI total = S x q x L + T0 + 36 x (H + A x r x L + Tm). L = $68.09; H = $48 + max(0, backup GB minus 10) x $0.00695. S and A are manual hours; q and r are AI shares; T0 and Tm are setup and monthly tool charges. Round totals only.

[Claude Pro](https://claude.com/pricing) and [ChatGPT Plus](https://chatgpt.com/pricing/) each cost $20/month. We charge $20 for a separate setup month, then $5/month for 36 operating months, allocating 25% only when other work uses the rest. Dedicated subscriptions add $540 to either total. Equal prices do not mean equal allowances; check limits before starting. API billing is separate.

Our cumulative API budgets are 200,000 to 1,000,000 setup input tokens and 20,000 to 100,000 output; monthly input is 100,000 to 500,000 and output 10,000 to 50,000, including retries and billed reasoning. Low input is 30% ordinary, 10% short cache writes and 60% reads; high input is uncached. Per million input/output/write/read tokens, [Opus](https://platform.claude.com/docs/en/about-claude/pricing) costs $4/$20/$5/$0.20; [Astra](https://developers.openai.com/api/docs/pricing) costs $10/$50/$12.50/$1. Writes replace ordinary input charges. API cost sums each token category times its rate, divided by 1,000,000.

Claude Code or Codex can prepare Spark installation commands and check the scheduled report configuration. Desktop Commander, on the server or through authorized SSH, can install dependencies, edit Spark settings and read failed-job logs. Its local server is free; optional Remote Pro costs $20/month, adding $200 at this allocation. Claude in Chrome can inspect the provider panel, Cloudflare records and Spark dashboard while we watch. ChatGPT Work uses a separate cloud browser where available; confirm enabled browser access.

The conclusion stays: EMR Serverless at $185 to $739 over 36 months remains cheaper. High-case subscriptions even exceed manual operation because monthly labor does not fall.

Use a new server or staging copy, approve commands or restrict a sandbox, and keep secrets out of chat. Watch dashboard actions and use subscriptions for supported person-directed tools, API keys for custom automated pipelines. Review changes and restore a report backup before relying on it; you still own access, updates and recovery.

See the [Spark pilot discussion in the post](https://jwatte.com/blog/apache-big-data-lakehouse-small-business/#setup-time-when-claude-or-chatgpt-does-the-typing) for context.

### Filled Flink example

[Managed Flink](https://aws.amazon.com/managed-service-apache-flink/pricing/): 1 to 2 processing KPUs plus one orchestration KPU, 730 hours at $0.11/KPU-hour, plus 50 GB per processing KPU at $0.10/GB/month. One Medium self hosted pilot, 24 to 60 setup hours and 6 to 16 monthly hours; no high availability. Durable managed backup and input stream costs are excluded.

| Cost | Low labor case | High labor case |
|---|---:|---:|
| Setup | $1,634 | $4,085 |
| Monthly operations | $457 | $1,139 |
| Self hosted, 36 months | $18,080 | $45,081 |
| Alternative monthly | $166 | $251 |
| Alternative, 36 months | $5,962 | $9,032 |
| Savings from self hosting | Negative $12,118 | Negative $36,048 |
| Break even month | Never | Never |

### Filled DataFusion example

[Athena](https://aws.amazon.com/athena/pricing/): 1 to 5 TB scanned x $5/TB, excluding existing source storage. One Small server, 4 to 12 setup hours and 1 to 3 monthly hours. These are query-operation comparisons, not complete reporting-app budgets.

| Cost | Low labor case | High labor case |
|---|---:|---:|
| Setup | $272 | $817 |
| Monthly operations | $92 | $230 |
| Self hosted, 36 months | $3,598 | $9,082 |
| Alternative monthly | $5 | $25 |
| Alternative, 36 months | $180 | $900 |
| Savings from self hosting | Negative $3,418 | Negative $8,182 |
| Break even month | Never | Never |

## Sources

Sources and prices read October 29, 2026. Recheck project advisories through [ASF security](https://www.apache.org/security/) and the project links before installing. Apache License terms and third-party dependency terms both matter.

1. [Quick Start, Spark 4.2.0 Documentation](https://spark.apache.org/docs/latest/quick-start.html). Read October 29, 2026.
2. [Hardware Provisioning, Spark 4.2.0 Documentation](https://spark.apache.org/docs/latest/hardware-provisioning.html). Read October 29, 2026.
3. [First Steps, Apache Flink](https://nightlies.apache.org/flink/flink-docs-stable/docs/getting-started/local_installation/). Read October 29, 2026.
4. [Installation , Apache DataFusion main documentation](https://datafusion.apache.org/user-guide/cli/installation.html). Read October 29, 2026.
5. [Ambari Installation Guide, Apache Ambari](https://ambari.apache.org/docs/3.0.0/quick-start/installation-guide/). Read October 29, 2026.
6. [Quickstart](https://amoro.incubator.apache.org/quick-start/). Read October 29, 2026.
7. [raw.githubusercontent.com, InstallationInstruction.md](https://raw.githubusercontent.com/apache/atlas/master/docs/src/documents/Setup/InstallationInstruction.md). Read October 29, 2026.
8. [apache/bigtop on GitHub](https://github.com/apache/bigtop). Read October 29, 2026.
9. [Overview, Apache Celeborn™](https://celeborn.apache.org/docs/latest/deploy/). Read October 29, 2026.
10. [Embedded Mode Prerequisites, Apache Drill](https://drill.apache.org/docs/embedded-mode-prerequisites/). Read October 29, 2026.
11. [Apache Fluo, Large-scale Incremental Processing](https://fluo.apache.org/). Read October 29, 2026.
12. [Real-Time Analytics with Flink, Apache Fluss™](https://fluss.apache.org/docs/quickstart/flink/). Read October 29, 2026.
13. [Guide, GeaFlow](https://geaflow.incubator.apache.org/docs/guide). Read October 29, 2026.
14. [Apache Gravitino, Apache Gravitino](https://gravitino.apache.org/docs/1.3.1/). Read October 29, 2026.
15. [Apache Hadoop 3.5.0 , Hadoop: Setting up a Single Node Cluster.](https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/SingleCluster.html). Read October 29, 2026.
16. [QuickStarted](https://hive.apache.org/development/quickstart/). Read October 29, 2026.
17. [Spark Quick Start, Apache Hudi](https://hudi.apache.org/docs/quick-start-guide). Read October 29, 2026.
18. [Impala Requirements](https://impala.apache.org/docs/build/html/topics/impala_prereqs.html). Read October 29, 2026.
19. [Getting Started , Apache Kyuubi](https://kyuubi.readthedocs.io/en/v1.12.0/quick_start/quick_start.html). Read October 29, 2026.
20. [Stand-alone deployment, Apache Linkis](https://linkis.apache.org/docs/latest/deployment/deploy-quick). Read October 29, 2026.
21. [Livy, Getting Started](https://livy.apache.org/get-started). Read October 29, 2026.
22. [Try Ozone With Docker, Apache Ozone](https://ozone.apache.org/docs/quick-start/installation/docker). Read October 29, 2026.
23. [raw.githubusercontent.com, quick start.mdx](https://raw.githubusercontent.com/apache/paimon/master/docs/docs/flink/quick-start.mdx). Read October 29, 2026.
24. [Getting Started](https://pig.apache.org/docs/r0.18.0/start.html). Read October 29, 2026.
25. [Configuring Polaris for Production, Apache Polaris](https://polaris.apache.org/releases/1.8.0/configuration/configuring-polaris-for-production/). Read October 29, 2026.
26. [Samza, Quick Start](https://samza.apache.org/startup/quick-start/latest). Read October 29, 2026.
27. [Quickstart Guide, Docker , incubator-sdap-nexus 1.0.0-SNAPSHOT documentation](https://incubator-sdap-nexus.readthedocs.io/en/latest/quickstart.html). Read October 29, 2026.
28. [Setting up a Storm Cluster](https://storm.apache.org/releases/current/Setting-up-a-Storm-cluster.html). Read October 29, 2026.
29. [Installation, Apache StreamPark](https://streampark.apache.org/docs/get-started/installation). Read October 29, 2026.
30. [Intro, Apache Uniffle](https://uniffle.apache.org/docs/intro). Read October 29, 2026.
31. [Incubation Policy](https://incubator.apache.org/policy/incubation.html). Read October 29, 2026.
32. [Apache Ambari](https://ambari.apache.org). Read October 29, 2026.
33. [ambari release release-3.0.0 on GitHub](https://github.com/apache/ambari/releases/tag/release-3.0.0). Read October 29, 2026.
34. [Apache Amoro™](https://amoro.incubator.apache.org/). Read October 29, 2026.
35. [Download](https://amoro.incubator.apache.org/download/). Read October 29, 2026.
36. [Apache Arrow, Apache Arrow](https://arrow.apache.org). Read October 29, 2026.
37. [projects.apache.org, projects.json](https://projects.apache.org/json/foundation/projects.json). Read October 29, 2026.
38. [Apache Atlas , Data Governance and Metadata framework for Hadoop](https://atlas.apache.org). Read October 29, 2026.
39. [Index of /atlas/2.5.0](https://downloads.apache.org/atlas/2.5.0/). Read October 29, 2026.
40. [Apache Auron™](https://auron.apache.org). Read October 29, 2026.
41. [auron release v8.0.0 on GitHub](https://github.com/apache/auron/releases/tag/v8.0.0). Read October 29, 2026.
42. [Apache Beam®](https://beam.apache.org). Read October 29, 2026.
43. [Beam Releases](https://beam.apache.org/get-started/downloads/). Read October 29, 2026.
44. [Bigtop , Apache Bigtop](https://bigtop.apache.org/). Read October 29, 2026.
45. [Index of /bigtop/bigtop-3.6.0](https://downloads.apache.org/bigtop/bigtop-3.6.0/). Read October 29, 2026.
46. [Apache Calcite • Dynamic data management framework](https://calcite.apache.org). Read October 29, 2026.
47. [Downloads](https://calcite.apache.org/downloads/). Read October 29, 2026.
48. [CarbonData](https://carbondata.apache.org). Read October 29, 2026.
49. [Index of /carbondata/2.3.2](https://downloads.apache.org/carbondata/2.3.2/). Read October 29, 2026.
50. [Apache Celeborn™](https://celeborn.apache.org). Read October 29, 2026.
51. [Download, Apache Celeborn™](https://celeborn.apache.org/download/). Read October 29, 2026.
52. [Apache DataFu](https://datafu.apache.org/). Read October 29, 2026.
53. [Index of /datafu/apache-datafu-2.1.0](https://downloads.apache.org/datafu/apache-datafu-2.1.0/). Read October 29, 2026.
54. [Apache DataFusion , Apache DataFusion main documentation](https://datafusion.apache.org). Read October 29, 2026.
55. [Index of /datafusion/datafusion-55.2.0](https://downloads.apache.org/datafusion/datafusion-55.2.0/). Read October 29, 2026.
56. [Apache Drill, Schema-free SQL for Hadoop, NoSQL and Cloud Storage](https://drill.apache.org). Read October 29, 2026.
57. [Download, Apache Drill](https://drill.apache.org/download/). Read October 29, 2026.
58. [Apache Flink® , Stateful Computations over Data Streams, Apache Flink](https://flink.apache.org). Read October 29, 2026.
59. [Downloads, Apache Flink](https://flink.apache.org/downloads/). Read October 29, 2026.
60. [Releases, Apache Fluo](https://fluo.apache.org/releases/). Read October 29, 2026.
61. [Apache Fluss™](https://fluss.apache.org). Read October 29, 2026.
62. [fluss release v1.0.0 on GitHub](https://github.com/apache/fluss/releases/tag/v1.0.0). Read October 29, 2026.
63. [GeaFlow](https://geaflow.incubator.apache.org/). Read October 29, 2026.
64. [Index of /incubator/geaflow/0.8.0-incubating](https://downloads.apache.org/incubator/geaflow/0.8.0-incubating/). Read October 29, 2026.
65. [Home, Apache Gluten](https://gluten.apache.org). Read October 29, 2026.
66. [gluten release v1.7.0 on GitHub](https://github.com/apache/gluten/releases/tag/v1.7.0). Read October 29, 2026.
67. [Welcome, Apache Gravitino](https://gravitino.apache.org). Read October 29, 2026.
68. [gravitino release v1.3.1 on GitHub](https://github.com/apache/gravitino/releases/tag/v1.3.1). Read October 29, 2026.
69. [Apache Hadoop](https://hadoop.apache.org/). Read October 29, 2026.
70. [Apache Hive](https://hive.apache.org). Read October 29, 2026.
71. [Downloads](https://hive.apache.org/general/downloads/). Read October 29, 2026.
72. [Apache Hudi, An Open Source Data Lake Platform, Apache Hudi](https://hudi.apache.org). Read October 29, 2026.
73. [Apache Iceberg, Apache Iceberg™](https://iceberg.apache.org). Read October 29, 2026.
74. [Impala](https://impala.apache.org). Read October 29, 2026.
75. [Apache Kyuubi, Multi-tenant Thrift JDBC/ODBC server](https://kyuubi.apache.org). Read October 29, 2026.
76. [kyuubi release v1.12.0 on GitHub](https://github.com/apache/kyuubi/releases/tag/v1.12.0). Read October 29, 2026.
77. [Apache Linkis, Apache Linkis](https://linkis.apache.org/). Read October 29, 2026.
78. [Release List, Apache Linkis](https://linkis.apache.org/download/main/). Read October 29, 2026.
79. [Apache Livy](https://livy.apache.org). Read October 29, 2026.
80. [Index of /incubator/livy/0.9.0-incubating](https://downloads.apache.org/incubator/livy/0.9.0-incubating/). Read October 29, 2026.
81. [Apache ORC • High-Performance Columnar Storage for Hadoop](https://orc.apache.org). Read October 29, 2026.
82. [Home, Apache Ozone](https://ozone.apache.org/). Read October 29, 2026.
83. [Downloads, Apache Ozone](https://ozone.apache.org/download/). Read October 29, 2026.
84. [Apache Paimon™](https://paimon.apache.org). Read October 29, 2026.
85. [paimon release release-2.1.0 on GitHub](https://github.com/apache/paimon/releases/tag/release-2.1.0). Read October 29, 2026.
86. [Parquet](https://parquet.apache.org). Read October 29, 2026.
87. [Welcome to Apache Pig!](https://pig.apache.org). Read October 29, 2026.
88. [Apache Pig Releases](http://pig.apache.org/releases.html). Read October 29, 2026.
89. [Apache Polaris](https://polaris.apache.org). Read October 29, 2026.
90. [Samza](https://samza.apache.org/). Read October 29, 2026.
91. [Apache SDAP, Science Data Analytics Platform](https://sdap.apache.org). Read October 29, 2026.
92. [Apache SDAP, Science Data Analytics Platform](https://sdap.apache.org/downloads). Read October 29, 2026.
93. [Redirecting…](https://sedona.apache.org). Read October 29, 2026.
94. [sedona release sedona-1.9.1 on GitHub](https://github.com/apache/sedona/releases/tag/sedona-1.9.1). Read October 29, 2026.
95. [Apache Spark™, Unified Engine for large-scale data analytics](https://spark.apache.org). Read October 29, 2026.
96. [Downloads, Apache Spark](https://spark.apache.org/downloads.html). Read October 29, 2026.
97. [Apache Storm](https://storm.apache.org/). Read October 29, 2026.
98. [Apache StreamPark, Apache StreamPark](https://streampark.apache.org). Read October 29, 2026.
99. [Apache StreamPark](https://streampark.apache.org/download/). Read October 29, 2026.
100. [Apache Tez , Welcome to Apache TEZ®](https://tez.apache.org). Read October 29, 2026.
101. [Home](https://toree.incubator.apache.org/). Read October 29, 2026.
102. [Index of /incubator/toree/0.5.0-incubating/toree](https://downloads.apache.org/incubator/toree/0.5.0-incubating/toree/). Read October 29, 2026.
103. [Home, Apache Uniffle](https://uniffle.apache.org). Read October 29, 2026.
104. [uniffle release v0.10.0 on GitHub](https://github.com/apache/uniffle/releases/tag/v0.10.0). Read October 29, 2026.
105. [Apache Wayang, Cross-Platform Data Processing](https://wayang.apache.org). Read October 29, 2026.
106. [Index of /wayang/1.1.1](https://downloads.apache.org/wayang/1.1.1/). Read October 29, 2026.
107. [Apache XTable™ (Incubating)](https://xtable.incubator.apache.org/). Read October 29, 2026.
108. [incubator-xtable release 0.4.0-incubating on GitHub](https://github.com/apache/incubator-xtable/releases/tag/0.4.0-incubating). Read October 29, 2026.
109. [Occupational Employment and Wage Statistics Profiles](https://data.bls.gov/oesprofile/?areas=INDUSTRY,STATE,MSA&major_group=150000&measure=01&occupation=151244&year=2025). Read October 29, 2026.
110. [Bureau of Labor Statistics Data](https://data.bls.gov/timeseries/CMU2020000000000P). Read October 29, 2026.
111. [Droplet Pricing, DigitalOcean](https://www.digitalocean.com/pricing/droplets). Read October 29, 2026.
112. [B2 Cloud Storage Pricing, Backblaze](https://www.backblaze.com/cloud-storage/pricing). Read October 29, 2026.
113. [Amazon EMR Pricing](https://aws.amazon.com/emr/pricing/). Read October 29, 2026.
114. [Amazon Managed Service for Apache Flink Pricing](https://aws.amazon.com/managed-service-apache-flink/pricing/). Read October 29, 2026.
115. [Amazon Athena Pricing](https://aws.amazon.com/athena/pricing/). Read October 29, 2026.
116. [ASF Security Team, Apache Software Foundation](https://www.apache.org/security/). Read October 29, 2026.

117. [Claude pricing](https://claude.com/pricing). Observed 2026-10-29 23:31 UTC; supplied October 29, 2026 research.
118. [ChatGPT pricing](https://chatgpt.com/pricing/). Observed 2026-10-29 about 23:27 UTC; supplied October 29, 2026 research.
119. [Claude API pricing](https://platform.claude.com/docs/en/about-claude/pricing). Observed 2026-10-29 23:31 UTC; supplied October 29, 2026 research.
120. [OpenAI API pricing](https://developers.openai.com/api/docs/pricing). Observed 2026-10-29 23:31 UTC; supplied October 29, 2026 research.
121. [OpenAI prompt caching](https://developers.openai.com/api/docs/guides/prompt-caching). Observed 2026-10-29 23:32 UTC; supplied October 29, 2026 research.
122. [Codex CLI](https://developers.openai.com/codex/cli). Observed 2026-10-29 23:32 UTC; supplied October 29, 2026 research.
123. [Using Codex with your ChatGPT plan](https://help.openai.com/en/articles/11369540-using-codex-with-your-chatgpt-plan). Observed 2026-10-29 23:31 UTC; supplied October 29, 2026 research.
124. [Desktop Commander pricing](https://desktopcommander.app/pricing/). Observed 2026-10-29 23:31 UTC; supplied October 29, 2026 research.
125. [Desktop Commander documentation](https://raw.githubusercontent.com/wonderwhy-er/DesktopCommanderMCP/main/README.md). Observed 2026-10-29 23:31 UTC; supplied October 29, 2026 research.
126. [Claude in Chrome setup](https://support.claude.com/en/articles/12012173-get-started-with-claude-in-chrome). Observed 2026-10-29 23:31 UTC; supplied October 29, 2026 research.
127. [Claude in Chrome permissions](https://support.claude.com/en/articles/12902446-claude-in-chrome-permissions-guide). Observed 2026-10-29 23:31 UTC; supplied October 29, 2026 research.
128. [ChatGPT Work cloud browser](https://help.openai.com/en/articles/20001280-using-cloud-browser-in-chatgpt). Observed 2026-10-29 23:32 UTC; supplied October 29, 2026 research.
129. [Anthropic Consumer Terms](https://www.anthropic.com/legal/consumer-terms). Observed 2026-10-29 23:31 UTC; supplied October 29, 2026 research.
130. [OpenAI Terms of Use](https://openai.com/policies/terms-of-use/). Observed 2026-10-29 about 23:27 UTC; supplied October 29, 2026 research.

*Apache, the Apache feather logo and the Apache project names are trademarks of The Apache Software Foundation. This site is not affiliated with or endorsed by the ASF. Other product names are used for identification only.*