How can I troubleshoot an active short circuit?

What are some effective methods of diagnosing an active short circuit? By this I mean a short circuit that presents itself only after a PCB is powered on.

tl;dr

I have a design in the prototype phase. 17 of my 20 boards work great. The other 3 all have a short circuit on a 3.3V rail. This only shows up after the board is powered on. After removing most of the components on the rail, I tracked it down to an Ethernet PHY. If I lift the IC off, my rail is rock-solid at 3.3V. When I put it back on (also tried 2 new ICs), my rail is overloaded again and it drops out.

I have thoroughly visually inspected, and probed the board for shorts, but cannot find one. I have lifted off pretty much everything around the IC (crystal, series resistors, ferrite beads, etc.) but still get the same behaviour. I've also tried holding the chip in reset, but that doesn't help. I lifted individual pins on the IC (VDDIO) and that fixed it, but doesn't offer a real diagnosis. I'm starting to wonder if there's an issue with the PCB fab, but not totally sure how it would cause this. They claim to do 100% E-test. Any advice will be appreciated!

Answer

While I wish you best of luck with your thermal camera endeavour, I would actually expect that the camera will show you the IC affected by the short (the Ethernet PHY?), not the short itself. You'd have to be pretty lucky for the actual faulty spot to have higher resistance than the internals of an IC.

If you find nothing with the camera, I would then check continuity to GND / 3.3V rail on any of the IC pins which are not actually connected to GND / 3.3V. Do it with a diode tested, since a forward-biased junction is close enough to a short.

If that comes out negative, I would power the PCB with the IC removed and check the voltage on all pins (ideally, a power-up waveform but it may be tedious for a large number of pins). Any voltage outside of 0..VCC range could potentially cause the IC to latch up, a condition which typically looks like a short.

Finally, I would check if all pads corresponding to the output pins can be actively driven. This can be done by connecting a scope and a signal generator together outputting a 0-3.3V square wave (so that you see the square wave on the scope), then connecting the probe to the pads. A disappearing square wave would mean that something else is trying to drive pad that the IC will want to drive as well. This can be justified for open-drain and bidir pins, but not for pure outputs.