Someone once told me that we are not supposed to worship on Sunday, but on Saturday. When I asked him how could that be since the New Testament clearly shows the change from Saturday to Sunday as the new "Christian Sabbath". His answer was, "the Apostles did that, but it wasn't what Jesus wanted". Wow! The Apostles, who were given the power of divine revelation, and inspired to speak perfect truth, made a massive mistake and no one before the 20th century caught it?! Really?!
There are so many places in Scripture that show that the Apostles had Jesus' full authority to lead the Church, that I do not have room to list them all here. One of the most clear references is the gospel of John 15:20 where Jesus says to the Apostles, "if they kept my word, they will keep yours also". He makes it fairly clear that those who want to obey Jesus will obey the Apostles. It is not as though the Apostles necessarily have to run around and convince people that they are the rightful leaders of the Church; Jesus says that people who want His truth will be drawn to the Church that the Apostles set up.
There is an automatic connection between Jesus' person, and His Church. We might see only bad things in the Church, and compare that to the good of our Lord, but that is misguided. If Jesus says that His followers will follow the Apostles, then that is the truth (regardless of whether we like it or not). He does not adapt the truth to our situation; we are supposed to adapt our situation to His truth. This is why it has always been the Catholic teaching that to reject the Church is to reject Christ. Yes, there are many who do so in ignorance, but that does not make it a good thing.
We must look at all those who turn from the Church that Jesus founded, and seek to bring them into the fullness of the faith -- even if some Catholic tells you otherwise. We are not to leave a fallen world in its blindness; we are supposed to show it the love of Christ and lead it to eternal salvation. Many people who want to follow Jesus are ignorant of His Church because they have been lied to by those who should know better. Jesus' words should give us confidence in this: if they really do want to follow Jesus, and we help to show them how that is done in His Church, then they will do so. Jesus promises it.