timing - What limits the lower bound of DRAM CAS latency

When a dram module receives a read/write command (while a row is active) it needs to:

1. 
   

   decode the command along with bank and column.

   
   
   


2. 
   

   multiplex the bank and send command along.

   
   


3. 
   

   (when reading) move the data from the column's latches to the send buffer ready to shift out after the CL expires, barrel shifting along the way according to the column address.

   
   

   (when writing) associate the relevant places in the receive buffer to he correct bank and barrel shift.

   
   

All this together seems to take a consistent 10 ns on modern DDRx dram modules while there is a clock that goes 10 times faster.

What is the biggest bottleneck in this sequence and could it be improved significantly or is there something else I'm missing here?