Would anyone like to explain why it remains the preferred protocol
Legacy.
BGP is a policy mechanism and sometimes the policies are either misconfigured or we can't trust hostile actors with access to wide open Internet (e.g. backwards countries null routing all of YouTube because censorship and it propagates outwards instead of inwards).
In most cases, if you are talking BGP to your ISP, your ISP filters your BGP traffic to only allow specific routes you can claim ownership of to be updated. Normally, non-infrastructure-level villains can't do bad BGP things if they have responsible upstream ISPs doing filtering correctly (kill you on flapping, kill routes you shouldn't be originating, etc). But, as we've seen with ISPs not even verifying UDP source address spoofing that allows you to generate multi-hundred-gigabit DDoS attacks, many ISPs are still run by morons.
BGP is also the magic behind anycast since you can intentionally duplicate any routes with no oversight (besides any upstream filtering in place).
what improvements are in the works
Good luck upgrading every embedded peering router in the world?
The goal of the Internet is no central point of failure. The downside (from a regulating abuse perspective) is there's no central point of authority either.
SMTP had the problem where any node on the Internet could send email for any other node on the Internet creating a game of N^2 whack-a-relay.
To even get access to the core Internet BGP peering infrastructure you have to be at the upper levels of ISP connectivity to start with. So, in the US at least, that requires maybe dropping a few thousand dollars and having Official Contacts first before you're even in the game of getting your own BGP peering arrangement.
If you are an intentional bad actor with bad actor connections like these awful italian hacker people then the only solution is after-the-fact punishment. The same goes if you are a country-level ISP (or any ISP part of the "core" Internet with no further upstream provider) and want to be a bad actor, then there's no oversight except when the rest of the world's network administrators comes together after seeing your malicious behavior and collectively say essentially "don't let Pakistan advertise any AS for Google properties."
(alternative answer: the internet should be based on the blockchain! Imagine if every network administrator had to get on /r/InternetBackbone at the same time to agree to shut down the Internet for 20 minutes so they can all deploy a bugfix to core-internet.exe. "uh oh, we accidentally forked the Internet again.")
It would absolutely be possible to have a functioning decentralized protocol. Look at the efforts like cjdns and snow. With the addition of some kind of lightweight payment ledger, you might be able to eliminate ISPs and switch to a model where people "mine bandwidth", generating revenue by switching on networking equipment running the protocol. Not saying it's easy, of course.
Legacy.
BGP is a policy mechanism and sometimes the policies are either misconfigured or we can't trust hostile actors with access to wide open Internet (e.g. backwards countries null routing all of YouTube because censorship and it propagates outwards instead of inwards).
In most cases, if you are talking BGP to your ISP, your ISP filters your BGP traffic to only allow specific routes you can claim ownership of to be updated. Normally, non-infrastructure-level villains can't do bad BGP things if they have responsible upstream ISPs doing filtering correctly (kill you on flapping, kill routes you shouldn't be originating, etc). But, as we've seen with ISPs not even verifying UDP source address spoofing that allows you to generate multi-hundred-gigabit DDoS attacks, many ISPs are still run by morons.
BGP is also the magic behind anycast since you can intentionally duplicate any routes with no oversight (besides any upstream filtering in place).
what improvements are in the works
Good luck upgrading every embedded peering router in the world?