Magic could operate differently from electromagnetism, but still interfere, such as with quantum effects. Inference doesn’t need to go both ways.

I thought about writing a magic setting with fairly hard justification for magic, and in my world, you’d control individual atoms and combine them to get the effects you want. You’d do this by gaining the respect of or instilling fear into atoms so they’d do your bidding. Spoken spells are more like tricks taught to dogs than having any power of their own, and the power derives from the respect or fear the atoms have for the caster. This explains why some wizards/witches are more powerful than others, and why learning isn’t necessarily the best way to get more powerful. The strongest magic users in my world spend a lot of time meditating, meaning communing with the target group of atoms.

The inner workings of atoms is poorly understood, so I think there’s room to insert some form of sentience.