In the absence of intellectual property, people could still distribute
binaries without the sources.
Yes, you are right. Therefore, simply eliminating copyright would not
make all software free. To do that, we would also need a consumer
protection law requiring publishers of software to make source code
available.