SEO and credit scores are not all that different. They're both pretty mysterious, and they both have many factors that contribute to them. One of the most important factors for both is length of history.
I am fortunate that my parents instilled the importance of a good credit score in me at a young age. Because of this, I got a credit card in high school. I used it for the basics (gas, shopping, movies) and paid it off every month. Because I started building credit when I was a teenager, I now have a solid length of credit history in my twenties. This contributes to my excellent credit score, which has allowed me to get a better mortgage rate, have easy access to financing, etc. If I were to start building credit just a few years ago, I would be starting from a much different starting line, and it would take more time, work, and effort to build my credit score to what it is today.
SEO is no different. The earlier you start investing in building your website’s relevancy, trust, and technical merit, the easier it will be to build that strong foundation in your search rankings. The longer you work at it, the more solidly grounded you are in your rankings. And just like with a credit score, if you stop working at it (cancel all your credit cards, stop bothering to pay bills on time), your score will dip.
SEO, like a credit score, is a marathon, not a sprint. But the sooner you start running, the faster you’ll get in the lead.