# Moving pandas and Excel reporting work to Polars

November 1, 2026. Companion to [Polars 2 for small and medium businesses](https://jwatte.com/blog/polars-2-small-business-guide/).

**Move one repetitive report first. Keep the old output until the new script agrees on the numbers that matter.** A successful migration gives someone a dependable report and a named person who can fix it when the export changes.

This kit covers pandas code and Excel or Power Query work that loads tables, cleans fields, joins reference lists and produces totals. It does not convert an entire workbook, its formulas, buttons, formatting or permission rules. We ran the examples with Python 3.12.15, pandas 3.1.0 and Polars 2.1.0 on synthetic data on November 1, 2026. This is a correctness exercise, not a speed benchmark or a production deployment.

## Choose a report worth moving

| Situation | Keep the current approach | Try Polars |
|---|---|---|
| A small report is easy to refresh and check | Excel or Power Query may already be the best fit | Only if repeated manual steps cause errors |
| A pandas script works and finishes on time | Keep it unless there is a measured problem | Test the slow transformation, not every script |
| Staff copy a folder of exports into a master sheet every week | Keep Excel as the review surface | Let a script build the reviewed table |
| Large scans, joins and grouped calculations dominate the job | Measure the current runtime and memory first | Test a lazy query against the same files |
| A specialist package expects pandas | Keep its required input type | Convert to pandas once at that boundary |
| Nobody can maintain Python | Keep a supported packaged tool | Arrange an owner and support before migrating |
| A workbook includes judgment, approvals or exception notes | Preserve the human decision step | Automate only the repeatable preparation |

The [Polars migration guide](https://docs.pola.rs/user-guide/migration/pandas/) explains the differences in execution and expressions. Its general performance claims do not establish a saving for your report. Time your own complete process, including input downloads, file writing and review. An export that takes twenty minutes to arrive will still take twenty minutes to arrive.

Write down the input owner, file format, report recipient, due time and allowed failure behavior. Decide whether a failed run should leave yesterday's report visible with its date or remove it from the handoff folder. An apparently fresh report with yesterday's totals is a particularly poor outcome.

## Translate the concepts before the syntax

| pandas or spreadsheet idea | Polars approach | Migration check |
|---|---|---|
| Index, including a MultiIndex | Keep keys in ordinary named columns | Materialize meaningful index values before converting |
| Label alignment during arithmetic | Expressions operate on columns in a table | Join on explicit business keys before combining tables |
| Mutation or an `inplace` option | Assign the returned frame from `with_columns`, `rename` or `drop` | Do not assume a method changed the original variable |
| Row `apply` or copied cell formulas | Use native column expressions | A Python callback often gives away the benefit of switching |
| `groupby` | `group_by(...).agg(...)` | Decide how null groups and output order should behave |
| `merge` | `join` | Set join type, key types and relationship validation |
| Pivot table | `pivot` with explicit aggregation | Confirm missing combinations and output column order |
| `to_datetime` and `.dt` | `.str.to_date` or `.str.to_datetime`, then `.dt` | Set format, time zone and business day boundaries |
| `NaN`, `None`, `pd.NA` | Distinguish null from floating point NaN | `fill_null` and `fill_nan` do different work |
| `.str` methods | `.str` expressions | Check regex behavior and whether trimming is intentional |
| `read_csv` | `read_csv` for an immediate frame or `scan_csv` for a query | Define types for identifiers and money |
| `to_csv`, `to_parquet`, `to_excel` | `write_csv`, `write_parquet`, `write_excel` | Keep the recipient's format and sheet names stable |
| Power Query applied steps | A sequence of expressions and joins | Preserve each business rule in a readable step |

Polars has no pandas style index. If your pandas index contains an invoice number, use `reset_index` to put it in a named column before conversion. Do not preserve an arbitrary row counter as if it were a customer identifier. [The conversion reference](https://docs.pola.rs/api/python/stable/reference/api/polars.from_pandas.html) documents `include_index` and its limits.

Null handling deserves a separate review. [pandas merge](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.merge.html) matches null keys to one another; [Polars join](https://docs.pola.rs/api/python/stable/reference/dataframe/api/polars.DataFrame.join.html) does not by default. You can explicitly select `nulls_equal=True` when that is the intended rule. For customer or product identifiers, rejecting a missing key is often a better business rule than joining unknown records together. Neither a matching grand total nor successful execution proves that the right customers received the right balances.

## Prepare an isolated trial

Create a new project folder and a virtual environment. Keep original exports read only and use synthetic data for this tutorial. The commands below install the exact versions used for these examples. They do not alter system Python.

On macOS or Linux:

```sh
python3 -m venv .venv
.venv/bin/python -m pip install --only-binary=:all: polars==2.1.0 pandas==3.1.0 pyarrow==26.0.0 xlsxwriter==3.3.1 openpyxl==3.1.5
```

On Windows PowerShell:

```powershell
py -m venv .venv
.venv/Scripts/python -m pip install --only-binary=:all: polars==2.1.0 pandas==3.1.0 pyarrow==26.0.0 xlsxwriter==3.3.1 openpyxl==3.1.5
```

Save the imports and fixture below at the top of a Python script. Then copy a pair of snippets after it. Each pair starts from the original `pd_orders` and `pl_orders`; it does not depend on the previous pair. The pandas result is called `p_result` and the Polars result `q_result`. Print them or inspect them in a notebook. The file examples create files in the current project folder; use an empty trial folder.

```python
import pandas as pd
import polars as pl

# All rows are synthetic examples, not customer records.
rows = {
    "order_id": [1, 2, 3, 4],
    "location": ["East", "West", "East", "West"],
    "product": ["Loaf", "Loaf", "Roll", "Roll"],
    "qty": [2, 1, 5, 3],
    "price_cents": [450, 450, 125, 125],
    "date_text": ["2026-10-01", "2026-10-01", "2026-10-02", "2026-10-02"],
    "customer": [" ALPHA ", "beta", " ALPHA ", "gamma"],
}
pd_orders = pd.DataFrame(rows)
pl_orders = pl.DataFrame(rows)
pd_prices = pd.DataFrame({"product": ["Loaf", "Roll"], "category": ["Bread", "Bread"]})
pl_prices = pl.DataFrame({"product": ["Loaf", "Roll"], "category": ["Bread", "Bread"]})
```

## Eighteen before and after examples

The test compared column names, row order and values for every pair. Integer cents are used for these synthetic prices. Data type equality is a separate production check because the libraries use different type systems.

### M01: Select columns

Keep only the fields the report needs.

pandas:

```python
p_result = pd_orders[["order_id", "qty"]]
```

Polars:

```python
q_result = pl_orders.select("order_id", "qty")
```

### M02: Filter rows

Keep orders with at least three units.

pandas:

```python
p_result = pd_orders.loc[pd_orders["qty"] >= 3, ["order_id", "qty"]]
```

Polars:

```python
q_result = pl_orders.filter(pl.col("qty") >= 3).select("order_id", "qty")
```

### M03: Create a money column

Integer cents avoid a floating point rounding decision in this example.

pandas:

```python
p_result = pd_orders.assign(total_cents=pd_orders["qty"] * pd_orders["price_cents"])[["order_id", "total_cents"]]
```

Polars:

```python
q_result = pl_orders.with_columns((pl.col("qty") * pl.col("price_cents")).alias("total_cents")).select("order_id", "total_cents")
```

### M04: Replace a row function

Express an arithmetic calculation directly instead of translating apply into a Python callback.

pandas:

```python
p_result = pd_orders.assign(total_cents=pd_orders.apply(lambda row: row["qty"] * row["price_cents"], axis=1))[["order_id", "total_cents"]]
```

Polars:

```python
q_result = pl_orders.select("order_id", (pl.col("qty") * pl.col("price_cents")).alias("total_cents"))
```

### M05: Set a conditional flag

All expressions must be valid for their inputs; do not use when as a guard around an invalid cast.

pandas:

```python
p_result = pd_orders[["order_id"]].assign(size=pd_orders["qty"].map(lambda n: "bulk" if n >= 3 else "regular"))
```

Polars:

```python
q_result = pl_orders.select("order_id", pl.when(pl.col("qty") >= 3).then(pl.lit("bulk")).otherwise(pl.lit("regular")).alias("size"))
```

### M06: Group totals by location

Sort explicitly so report order is part of the contract.

pandas:

```python
p_result = pd_orders.groupby("location", as_index=False)["qty"].sum().sort_values("location")
```

Polars:

```python
q_result = pl_orders.group_by("location").agg(pl.col("qty").sum()).sort("location")
```

### M07: Join a product list

Validate a many to one relationship so duplicate product keys cannot inflate sales.

pandas:

```python
p_result = pd_orders.merge(pd_prices, on="product", how="left", validate="many_to_one")[["order_id", "category"]].sort_values("order_id")
```

Polars:

```python
q_result = pl_orders.join(pl_prices, on="product", how="left", validate="m:1").select("order_id", "category").sort("order_id")
```

### M08: Fill missing quantities

These examples explicitly choose zero. That choice needs business approval for real data.

pandas:

```python
p_result = pd.DataFrame({"qty": pd.Series([2, None, 5], dtype="Int64")}).fillna({"qty": 0})
```

Polars:

```python
q_result = pl.DataFrame({"qty": [2, None, 5]}).with_columns(pl.col("qty").fill_null(0))
```

### M09: Parse dates

The input is year, month, day. Both sides reject invalid dates.

pandas:

```python
p_result = pd_orders[["order_id"]].assign(date=pd.to_datetime(pd_orders["date_text"], format="%Y-%m-%d").dt.strftime("%Y-%m-%d"))
```

Polars:

```python
q_result = pl_orders.select("order_id", pl.col("date_text").str.to_date("%Y-%m-%d").dt.strftime("%Y-%m-%d").alias("date"))
```

### M10: Normalize customer text

Whitespace and case cleaning is useful for comparison, but is not proof that two customers are the same person.

pandas:

```python
p_result = pd_orders[["order_id"]].assign(customer=pd_orders["customer"].str.strip().str.lower())
```

Polars:

```python
q_result = pl_orders.select("order_id", pl.col("customer").str.strip_chars().str.to_lowercase())
```

### M11: Sort a report

Use a second key to make ties deterministic.

pandas:

```python
p_result = pd_orders[["order_id", "qty"]].sort_values(["qty", "order_id"], ascending=[False, True])
```

Polars:

```python
q_result = pl_orders.select("order_id", "qty").sort(["qty", "order_id"], descending=[True, False])
```

### M12: Deduplicate values

This produces a distinct product list; it does not decide which conflicting customer record wins.

pandas:

```python
p_result = pd_orders[["product"]].drop_duplicates().sort_values("product")
```

Polars:

```python
q_result = pl_orders.select("product").unique().sort("product")
```

### M13: Stack export batches

The two parts have identical columns and types.

pandas:

```python
p_result = pd.concat([pd_orders.iloc[:2], pd_orders.iloc[2:]], ignore_index=True)
```

Polars:

```python
q_result = pl.concat([pl_orders.head(2), pl_orders.tail(2)], how="vertical")
```

### M14: Pivot a location table

Choose the aggregation and output column order explicitly.

pandas:

```python
p_result = pd_orders.pivot_table(index="location", columns="product", values="qty", aggfunc="sum", fill_value=0).reset_index()[["location", "Loaf", "Roll"]]
p_result.columns.name = None
```

Polars:

```python
q_result = pl_orders.pivot(on="product", index="location", values="qty", aggregate_function="sum").select("location", "Loaf", "Roll").sort("location")
```

### M15: Attach group totals to each row

A window expression keeps the original row count.

pandas:

```python
p_result = pd_orders[["order_id"]].assign(location_qty=pd_orders.groupby("location")["qty"].transform("sum"))
```

Polars:

```python
q_result = pl_orders.select("order_id", pl.col("qty").sum().over("location").alias("location_qty"))
```

### M16: Read and summarize CSV

The setup below creates this synthetic input file. Polars scans it lazily before collecting the summary.

pandas:

```python
pd_orders.to_csv("orders.csv", index=False)
p_result = pd.read_csv("orders.csv", usecols=["location", "qty"]).groupby("location", as_index=False)["qty"].sum().sort_values("location")
```

Polars:

```python
pl_orders.write_csv("orders.csv")
q_result = pl.scan_csv("orders.csv").group_by("location").agg(pl.col("qty").sum()).sort("location").collect()
```

### M17: Save and reload Parquet

Each library writes its own file; the comparison checks the data that comes back.

pandas:

```python
pd_orders.to_parquet("pandas-orders.parquet", index=False)
p_result = pd.read_parquet("pandas-orders.parquet")
```

Polars:

```python
pl_orders.write_parquet("polars-orders.parquet")
q_result = pl.read_parquet("polars-orders.parquet")
```

### M18: Write an Excel handoff

Each side writes a workbook and reads it back. The Polars side selects the openpyxl engine for this tested example.

pandas:

```python
pd_orders[["order_id", "qty"]].to_excel("pandas-report.xlsx", sheet_name="Orders", index=False)
p_result = pd.read_excel("pandas-report.xlsx", sheet_name="Orders")
```

Polars:

```python
pl_orders.select("order_id", "qty").write_excel("polars-report.xlsx", worksheet="Orders")
q_result = pl.read_excel("polars-report.xlsx", sheet_name="Orders", engine="openpyxl")
```

## Keep a narrow bridge to pandas or Arrow

These three conversions were also checked against the synthetic fixture above.

```python
# M19: pandas to Polars. Make meaningful index values columns first.
converted = pl.from_pandas(pd_orders)
assert converted.equals(pl_orders)

# M20: Polars to pandas, retaining Arrow backed columns where possible.
back_to_pandas = pl_orders.to_pandas(use_pyarrow_extension_array=True)
assert back_to_pandas.to_dict("records") == pd_orders.to_dict("records")

# M21: pass through an Arrow table.
arrow_table = pl_orders.to_arrow()
back_from_arrow = pl.from_arrow(arrow_table)
assert back_from_arrow.equals(pl_orders)
```

The [to_pandas reference](https://docs.pola.rs/api/python/stable/reference/dataframe/api/polars.DataFrame.to_pandas.html) requires pandas and PyArrow. It documents copying by default and Arrow backed extension arrays as an option that can preserve null values and permit zero copy operations. Later pandas operations may still trigger conversion. Do not promise that an entire mixed library pipeline will avoid copies.

Keep pandas where a downstream interface needs it. For example, the [statsmodels formula guide](https://www.statsmodels.org/stable/example_formulas.html) describes its data argument as a pandas frame. [GeoPandas](https://geopandas.org/en/stable/docs/user_guide/data_structures.html) builds its spatial table types on pandas. A Polars DataFrame is not a replacement for GeoPandas geometry behavior. Convert at a deliberate boundary and test what that library actually accepts. Do not assume every plotting or machine learning package needs pandas; check the particular API and version you use.

Avoid converting after every calculation. Do the table preparation in one library, cross the boundary once, and leave the specialist analysis in the library built for it. Include conversion time and memory when measuring the trial.

## Replace one Excel macro or Power Query flow

For example, an eight person bakery with two locations might have a workbook that appends weekly exports, looks up product categories and refreshes a sales pivot. This is an illustration. Its staff would still review the report in Excel; the script would prepare the numbers.

First, list the workbook's rules in plain language. Which cancelled orders are removed? Is a refund negative revenue or a separate column? Is the reporting day based on the till's local time? Does the workbook contain manual adjustments? Keep those adjustments in a separately reviewed input table with a reason and author. Do not silently overwrite them when a script runs.

For the synthetic tutorial, our complete rule is smaller: multiply quantities by integer prices, attach one category per product, total cents by location and category, and write a workbook. Save the fixture from earlier and run this block after it:

```python
# M22: synthetic report flow, with a pandas baseline for comparison.
pl_orders.write_csv("orders.csv")
pd_baseline = (
    pd_orders.assign(total_cents=pd_orders["qty"] * pd_orders["price_cents"])
    .merge(pd_prices, on="product", how="left", validate="many_to_one")
    .groupby(["location", "category"], as_index=False)["total_cents"]
    .sum()
    .sort_values(["location", "category"])
)
clean = pl.scan_csv("orders.csv").with_columns(
    (pl.col("qty") * pl.col("price_cents")).alias("total_cents")
).collect()
assert clean.height == 4
assert clean["order_id"].n_unique() == 4
assert clean["product"].null_count() == 0
joined = clean.join(pl_prices, on="product", how="left", validate="m:1")
assert joined.height == clean.height
assert joined["category"].null_count() == 0
report = joined.group_by("location", "category").agg(
    pl.col("total_cents").sum()
).sort("location", "category")
assert report["total_cents"].sum() == 2350
assert report.to_dicts() == pd_baseline.to_dict("records")
report.write_excel("weekly-sales.xlsx", worksheet="Sales")
report.write_parquet("weekly-sales.parquet")
```

The expected rows are East, Bread, 1525 cents and West, Bread, 825 cents. The script deliberately collects before the validated join. This small tutorial makes the checks easy to inspect; do not treat it as a memory sizing recommendation for a large export.

Replace the synthetic CSV with a copied real export only after checking its column names, types and data handling requirements. For a folder of exports, inventory every file and date range before appending. The common mistake is including both a corrected export and the original, which doubles some sales. Add an explicit order identifier rule and an exceptions report before scheduling.

A script does not refresh Excel formulas or execute VBA merely because it can write an XLSX file. If the old workbook contains formulas or charts people rely on, decide whether the script supplies a separate data sheet, produces a new plain report, or leaves the workbook unchanged. Test the real handoff in Excel with the person who reads it.

## Test the change and keep a working way back

1. Freeze a representative input set. Include a quiet week, a busy week, returns, duplicate identifiers, missing prices and a month boundary. Keep the same input files for both runs. Avoid moving live customer data into a personal notebook or public code repository.
2. Save the old script, dependency versions and expected reports. Record what it does today, including known errors, so an intentional correction is not confused with a migration regression.
3. Compare counts before and after each filter and join. For a many to one lookup, assert the right side's key is unique and the output row count is unchanged. Count unmatched keys and missing required values.
4. Compare money in integer cents or an explicitly agreed decimal representation. Check totals by location, product and period, not just the grand total. Compare dates, time zones, schema and column order. Sort by stable business keys before comparing unordered output.
5. Test invalid input. A renamed header, malformed date, new currency or duplicate reference key should produce a useful failure or an explicit exception file. Never quietly turn every parsing failure into zero.
6. Run both versions through at least one normal reporting cycle before switching the recipient's handoff. This is a suggested process, not a guarantee; month end or seasonal logic needs its own representative run.
7. Pin the approved dependencies, record the interpreter version and keep a fresh environment recipe. Save a small synthetic regression fixture with expected output so an update can be checked without customer records.
8. Keep the old scheduled job disabled but recoverable until the owner accepts the replacement. If reconciliation fails, stop the new job, preserve its rejected output privately, run the old environment against the frozen inputs and notify the report owner. Do not let both jobs write the same destination.

Record runtime, peak memory and total review time for both paths. Count the time spent maintaining export mappings and investigating exceptions. A faster calculation that makes the report harder to trust is not an improvement.

## Sources and test record

All documentation below was read November 1, 2026. The synthetic examples passed with Python 3.12.15, pandas 3.1.0 and Polars 2.1.0. M01 through M18 compared values and ordered columns between both libraries. M19 through M21 checked conversion round trips. M22 checked the complete synthetic reporting flow against its pandas baseline. These tests do not certify untested customer data, optional engines or future releases.

- [Polars: coming from pandas](https://docs.pola.rs/user-guide/migration/pandas/), index, expressions, eager and lazy execution. Observed November 1, 2026, 17:55:15 UTC.
- [Polars missing data](https://docs.pola.rs/user-guide/expressions/missing-data/), null and NaN handling. Observed November 1, 2026, 17:55:20 UTC.
- [Polars from_pandas](https://docs.pola.rs/api/python/stable/reference/api/polars.from_pandas.html), conversion and index handling. Observed November 1, 2026, 17:55:16 UTC.
- [Polars to_pandas](https://docs.pola.rs/api/python/stable/reference/dataframe/api/polars.DataFrame.to_pandas.html), dependencies and Arrow backed arrays. Observed November 1, 2026, 17:55:18 UTC.
- [Polars pivot](https://docs.pola.rs/api/python/stable/reference/dataframe/api/polars.DataFrame.pivot.html), explicit aggregation and eager pivot API. Observed November 1, 2026, 17:55:19 UTC.
- [Polars join](https://docs.pola.rs/api/python/stable/reference/dataframe/api/polars.DataFrame.join.html), validation and null matching. Observed November 1, 2026, 17:57:13 UTC.
- [pandas API reference](https://pandas.pydata.org/docs/reference/index.html), transformation and file APIs. Observed November 1, 2026, 17:55:22 UTC.
- [pandas merge](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.merge.html), null key warning and relationship checks. Observed November 1, 2026, 17:57:13 UTC.
- [pandas comparison with spreadsheets](https://pandas.pydata.org/docs/getting_started/comparison/comparison_with_spreadsheets.html), spreadsheet concepts and table operations. Observed November 1, 2026, 17:55:23 UTC.
- [statsmodels formula guide](https://www.statsmodels.org/stable/example_formulas.html), pandas input for the documented formula interface. Observed November 1, 2026, 17:55:24 UTC.
- [GeoPandas data structures](https://geopandas.org/en/stable/docs/user_guide/data_structures.html), pandas based spatial tables. Observed November 1, 2026, 17:56:53 UTC.

For installation, scheduling and backups, use the [small business kit](https://jwatte.com/downloads/polars-small-business-kit.md). For an existing Polars 1.x application, use the separate [Polars upgrade guide](https://jwatte.com/downloads/polars-upgrade-guide.md).