When accepting a calendar invite from Google you might receive a DSN back, similar to the one shown below. 

This happens when your domain is DMARC protected ie. you are in a policy of reject. The reason why your replies are blocked is because when you accept the invitation using Google's response buttons your response is sent back through Google’s servers.
Google effectively sees an email with your domain in the From address but originating from their servers and rejects it because your DMARC policy says so.
​
​Google’s response buttons are shown below.

There should be another set of response buttons included in the email invitation called out-of-band buttons. An example of such buttons is shown below. 

Using the out-of-band buttons when responding works around the above problem. However, during our testing we found that not all mail clients display the out-of-band buttons. 

Below is an illustration of the mail flow showing what happens when you reply using the different response buttons.
An Office 365 domain is used as the invitation receiver. The process should be the same with other mail providers. However, if the out-of-band buttons are missing then another recommendation would be to respond back to the sender with an email.