MSP Vendors - Security and transparency has to come first, not politics.

![[Pasted image 20240504234441.png]] __So it's been an interesting July so far in the Managed Services world; our communities such as__ [__MSPGeek__](https://mspgeek.org/) __and__ [__WinAdmins__](https://aka.ms/winadmins) __grew further in numbers and quality, and there's now a bimonthly Tuesday night (UK time) meeting of MSP members on the MSPGeek video chat.__ Everything fell apart however as Kaseya VSA, burned to the ground on a Friday, whilst the entire MSP world watched in horror at a nightmare scenario playing out in reality. A subsection of that MSP world is still recovering today from ransomware deployed directly to their devices, using the very mechanism they chose and trusted to implement the core of their business model. The community has rallied together in a beautiful fashion and we have numerous MSPs supporting our industry and their competitors in recovering from everyone's worst nightmare. Let alone when it applies to concurrent disaster recovery for dozens of their customers, often with unique and varied setups in their own right. [I have to give a specific mention to the team at Huntress Labs who have been invaluable with supporting, gathering/analysing threat intel, and working with the community at large on updates around this situation - at no cost.](https://www.huntress.com/blog/rapid-response-kaseya-vsa-mass-msp-ransomware-incident) Unfortunately in the aftermath it became very apparent that several things were going wrong both with the recovery of VSA services, the story around the vulnerability, and the presentation of the situation by Kaseya. Sadly, it's disappointing to see an extended outage, and a large timeframe until services were restored over a week after the incident occurred. Having lost access to a core tool in the bag, the inability to serve customers in any form through their tools was an additional blow, especially with several missed deadlines. Approaching the situation with several fixes was the correct method, tainted and mired by the messages from the Kaseya team. --- Nadir - General Manager of ITGlue (owned by Kaseya) lost both his dignity and professionalism in a public forum, whilst making the biggest mistake possible on the internet. Declaring a platform 100% safe 👀 - Lets hope infosec twitter doesnt see this. ![[Pasted image 20240504234514.png]] Now, i can totally understand Connectwise's decision to suspend the integration and connectivity between their platform, and an entity which has been compromised. Until further information and external validation came forward in a measured timeframe its the appropriate reaction to take to safeguard the mutual customers. It is also an interesting comment overall - especially after years of neglect for the platform, the community and their support around it. Unlike beetlejuice however, you only need to potentially bad mouth Kaseya once to get a reaction - whilst legitimate concerns and questions are ignored. --- So what did we learn in the coming days? - [DIVD had communicated several vulnerabilities to Kaseya.](https://csirt.divd.nl/cases/DIVD-2021-00011/) - [Kaseya's best practice document excluded the working directories of the RMM software in antivirus software - opening users up to exploitation](https://searchsecurity.techtarget.com/news/252503633/Kaseya-ransomware-attacks-What-we-know-so-far#:~:text=Due%20to%20compatibility%20problems%20with%20some%20antivirus%20tools%2C,customer%20systems%20once%20the%20VSA%20server%20was%20compromised.) - Fixes were in progress. - [REvil exploited VSA before the patches could be completed.](https://www.bleepingcomputer.com/news/security/kaseya-was-fixing-zero-day-just-as-revil-ransomware-sprung-their-attack/) - [The security posture of Kaseya VSA has been limited for years according to former employees](https://www.bloomberg.com/news/articles/2021-07-10/kaseya-failed-to-address-security-before-hack-ex-employees-say) - [ITGlue stooped to new levels of faeces throwing](https://web.archive.org/web/20210710231943/https://www.itglue.com/alternatives-for-connectwise/) - [They have now changed their marketing and sales technique to drop the Kaseya name](https://www.reddit.com/r/msp/comments/oloj2p/kaseya_trying_to_capitalize_on_their_own_hack/) --- Offsetting some of the less than stellar approaches on how to do incident management and PR during this time, other vendors have taken a step by step, playbook based approach to supporting their customers, and communicating on their security postures. Datto and NinjaRMM have particularly come out of this with a well thought out, and consistent approach to layering defences to both limit the damage should an attack occur - and setting themselves up for maximising the ability to detect and respond to such incidents. They are the only two vendors who have approached the community and engaged in reassuring MSPs, communicated extensively to questions, as well as keeping their team in line on the sales front. Other vendors are still catching up to current best practice with their legacy code bases, even just to implement the basics such as a WAF in-front of the stack and IP restrictions. ![[Pasted image 20240504234527.png]] This attack could have happened to any provider of software, and the lines drawn by these vendors will be remembered in years to come. How can we as a community engage with providers who are constantly kicking each other when they are down? How can we trust what they say when it is just another stab in the back of a competitor? How can we trust their trust pages when they're all a fog of war? Overall, the community and myself are disappointed by the response of ITGlue/Kaseya/Thoma Bravo et al, we don't forget the stance, values, and organisational behaviour you espouse when you place the value of shareholders, twisted optics, and finger pointing above an absolute basic value; owning your issues, and fixing them. The hidden organisational behaviour of the sales teams values and upper management leaves much to be desired, and a bitter taste to those of us handing payments over monthly to a firm apparently operated by a fleet of monkey-operated typewriters. Unfortunately, this isn't the works of Shakespeare being created and shared; it's a web of toxic deflections and lack of ethical leadership. --- [[Epistemic status|Colophon: Brewing]]