Yes, but then IDM will not capture downloads from Chromium browsers. If you want only manual download links via right-click, deletion is fine.
If you are writing this for a technical blog or a school assignment, consider these three pillars: The Evolution of Downloading: Compare "native" browser downloads to "accelerated" ones. User Experience (UX): idmgcext.crx
Forensics and analysis steps for a CRX file Yes, but then IDM will not capture downloads