In-Memory Compute: How AI Moves the Maths to the Data

In-memory compute (IMC) does the arithmetic inside the memory array instead of moving data out to a processor. For AI inference, the weights stay in place, the input activations drive the wordlines, and each bitline adds up the products, so the multiply-accumulate happens where the data lives. It matters because moving a number costs far more energy than computing with it: in published 45 nm figures, a 32-bit DRAM read costs about 640 pJ against about 0.1 pJ for a 32-bit integer add.

Originally published on LinkedIn. This version adds the technical detail behind the argument.

Key takeaways

• The von Neumann split of memory and processor, set out in 1945, makes data movement the main energy cost of modern AI inference.
• In analog IMC, Ohm's law does the multiply and Kirchhoff's current law does the accumulate, on the bitline.
• The cost moves to the edges of the array: DACs, ADCs, sense amplifiers and calibration often dominate area and energy.
• SRAM IMC is the most mature; DRAM and non-volatile options (ReRAM, PCM, flash) offer density but bring variability, endurance and precision problems.
• The mapping compiler, not the array, decides whether an IMC chip is usable. Nobody has won this yet.

Computing broke the rule it was built on

The rule dates from 1945, from John von Neumann's draft report on the EDVAC: memory holds the data, the processor does the maths, and a bus shuttles between them. Eighty years of nearly every chip you have used obeyed that line. In-memory compute deletes it. The maths now happens inside the memory array.

I have watched this land in an architecture proposal. Someone puts up the energy breakdown for an edge inference workload, and the room goes quiet, because the compute was never the problem.

The number that should not be possible

Mark Horowitz's ISSCC 2014 keynote paper, "Computing's energy problem (and what we can do about it)", gave figures for a 45 nm process that are still the standard reference:

Operation (45 nm, Horowitz, ISSCC 2014)	Approximate energy
8-bit integer add	0.03 pJ
32-bit integer add	0.1 pJ
8-bit integer multiply	0.2 pJ
32-bit floating-point multiply	3.7 pJ
32-bit read from an 8 KB SRAM	5 pJ
32-bit read from DRAM	640 pJ

That is roughly 6,000 times more energy to move the number from DRAM than to use it in an add. Newer nodes have cut the logic energy faster than the off-chip access energy, so the ratio has, if anything, got worse since.

You cannot fix that with a better core. You cannot fix it with a smaller node. Wires do not scale the way transistors do, which is why the memory wall kept widening while Moore's Law was still delivering.

A worked example (illustrative)

Take a small edge model with 4 million 8-bit weights, so about 4 million multiply-accumulates (MACs) per inference. Using the table above:

• Arithmetic: 4M MACs x (0.2 + 0.03) pJ is about 0.9 uJ.
• Weights from DRAM every inference: 4 MB is 1M 32-bit words, x 640 pJ, about 640 uJ. Data movement is roughly 700 times the arithmetic.
• Weights held in on-chip SRAM: 1M reads x 5 pJ, about 5 uJ. Much better, but still about five times the arithmetic, and that is before activations and partial sums move.
• Weights held in an IMC array: no weight reads at all. The energy now goes into driving wordlines and converting bitline results. If, say, one 8-bit column conversion costs about 1 pJ and each column sums 256 products, conversion adds about 0.004 pJ per MAC per input bit. With 8-bit inputs applied one bit at a time, that becomes about 0.03 pJ per MAC.

These numbers are illustrative, chosen to show the shape of the trade-off, not to describe any product. The point stands: the old flow pays mostly for movement, and IMC moves the bill to the converters.

Old way and new way

So the industry stopped racing towards the finish line. It moved the finish line.

Step	Conventional accelerator	In-memory compute
1	Weights in DRAM	Weights stay in the array
2	Fetch to SRAM	Drive the wordlines with the inputs
3	Fetch to registers	Let the bitline do the multiply-accumulate
4	Multiply	Read out the result through the ADC
5	Write back, repeat millions of times	No fetch, no bus, no round trip

How the bitline does the maths

In a resistive crossbar (ReRAM or PCM), each weight is stored as a conductance G. Apply an input voltage V to a row and the cell passes a current I = G x V: Ohm's law does the multiply. All the cells on one column share a bitline, so their currents add: Kirchhoff's current law does the accumulate. The column current is the dot product of the input vector and that column's weights, computed in one step for every column at once.

SRAM IMC works on the same idea with different physics. The weight is a stored bit; the input gates a read path; the products appear as current, as charge on a shared capacitor, or as a digital count from an adder tree beside the array (often called digital IMC). Charge-domain designs are popular because capacitors match well across a die, which helps accuracy.

That one change rewrites the stack:

• Wordline drivers become the input layer.
• The bitline becomes the accumulator.
• Sense amplifiers become the arithmetic readout.
• ADCs and DACs become the new critical path.
• The compiler becomes the thing that decides whether any of it works at all.

Five layers. Not one of them is a CPU.

Which memory takes it

Technology	Strengths	Open problems
SRAM IMC (analog or digital)	Standard CMOS, any logic node, fast, unlimited write endurance, most mature	Low density, leakage, weights lost at power-off
DRAM IMC and processing-in-memory	High capacity, existing supply chain, suited to large models	DRAM process is poor for logic, refresh, thermal limits
ReRAM crossbar	Dense, non-volatile, natural analog multiply	Device variability, drift, limited write endurance, programming time
Phase-change memory (PCM)	Multi-level cells, non-volatile, well studied	Conductance drift, write energy, endurance
Flash (NOR) IMC	Mature, dense, non-volatile, multi-level storage	Slow, energy-hungry writes, limited endurance, older nodes

Public examples span the range. Samsung announced HBM-PIM, with compute units inside high-bandwidth memory, in 2021; SK hynix showed GDDR6-AiM; UPMEM sells DRAM with small processors on the die. IBM Research published an analog PCM inference chip in 2023 and, separately, NorthPole, a digital design that puts memory right beside compute. Start-ups including Mythic (analog flash), EnCharge AI (charge-domain SRAM) and d-Matrix (digital IMC) have each taken a different branch. These are as announced by the companies and in their papers, not independent benchmarks.

The real story: who owns the architecture

Here is the real story, and it has nothing to do with picojoules. In-memory compute quietly changes who owns the architecture. For fifty years memory was a commodity you bought, and logic was where you competed.

IMC inverts that. If the multiply happens on the bitline, then your memory technology, your device variability, your ADC design and your calibration scheme are your architecture.

• Memory vendors become compute vendors.
• Mixed-signal teams become the critical path.
• The moat moves out of the RTL and into the mapping compiler, because a crossbar nobody can program is just an expensive resistor.

The array is the Trojan horse. The software stack is the takeover. Stop moving data to the maths. Move the maths to the data.

What is still open

Nobody has won this yet. The problems still open:

• Variability. Cell-to-cell and die-to-die spread shows up directly as arithmetic error. Calibration and noise-aware training help, at a cost.
• ADC overhead. High-resolution converters on every column eat the energy saved. Designs share ADCs, cut resolution or go digital.
• Limited precision. Analog arrays typically give the equivalent of a few bits per operation. Larger precision needs bit-slicing over cells or cycles.
• Write endurance. Non-volatile cells wear out, which suits fixed inference weights but not training or frequent model swaps.
• Verification. Signing off a chip whose arithmetic is statistical needs new methods: accuracy at corners, not just functional correctness.
• Tool flow. Mapping layers onto tiles, scheduling, partial sums and calibration needs a compiler stack the industry has not finished building.

Common misconceptions

• "IMC removes the memory wall." It removes weight movement. Activations, partial sums and model changes still move.
• "Analog means imprecise, so it is useless." Many inference workloads tolerate low precision well, provided the model is trained or tuned for the hardware.
• "It is just near-memory compute." Near-memory puts logic beside the array; in-memory uses the array itself as the arithmetic unit. Both cut movement, by different amounts.

FAQ

What is in-memory compute in simple terms?

A memory array that also does arithmetic, so data does not have to travel to a processor before it is used.

Why is in-memory compute important for AI?

Neural network inference is dominated by multiply-accumulates on large, fixed weight sets. Keeping those weights in place removes most of the energy spent on moving them.

Is SRAM or ReRAM better for in-memory compute?

SRAM is mature and works on standard logic processes; ReRAM is denser and non-volatile but harder to make accurate and durable. The right choice depends on model size, precision needs and how often weights change.

What limits in-memory compute today?

Converter overhead, device variability, limited precision, endurance, verification methods and the compiler tool flow.

Your call

So, your call: which layer takes it? A mature SRAM IMC, high-density DRAM IMC, or the ReRAM crossbar?

And the more interesting question. Whoever wins the array, who wins the compiler?