Why would the Inner Party go to such lengths in the Ministry of Love and elsewhere unless many are defying the party, if only in small ways like the promiscuous Julia? Surely her very many sexual partners are also defying the Inner Party. Winston's holding cell in the Ministry of Love was full, but how many such cells are there in that huge building and how many equivalents of Room 101? The extent of unrest in the Outer Party is obvious from the time and effort spent in quelling it - even planting Charrington and installing and monitoring telescreens in prole territory.
Are the rest of the Outer Party 'converted'? Fear obviously inhibits any member from showing even a hint of defiance, and Orwell only gives us insight into the mindset of the small circle around Winston. Surely these few members are representative of the other six million. And only a minority resemble compliant and stupid Parsons in adoring the totalitarian regime.
Winston acts to defy the party because he is representative of a large, but shrinking, remnant who remember life before INGSOC. Significantly Winston is a dreamer and a fool because resistance is useless. Julia, and all her pragmatic peers, well understand that the Party is now invulnerable. They do what they can to make life more bearable with no thought of outright rebellion.
If Winston and Julia were "the only ones who acted on their impulses to deliberately defy The Party" and all others compliant zombies, then 1984 would be much less than the thoroughly grim book it is. The politics of 1984 are poison.

