The Federation committed a unspeakable crime removing Saru from his people. They interfered in the possible evolution of there entire culture, denying this culture of not only of a genius able to learn from alien technology, but one with rebellious thoughts, that they should stand up to the people who kidnap them, That they should take and learn from their technology. that they should reach out to the cosmos.
Think how human history would be different if someone took people like Leonardo da Vinci or galileo or Einstein from us.
This is the whole point of the prime directive, to prevent the federation from interfering with the natural progression of a species. His removal from his species has set it back decades, even centuries.