In summary - making a credit memo can increase the stock quantity but will never set the "Stock Availability" of the product to "In Stock", whereas a config option can control whether an order cancellation sets "In Stock" automatically. Denying a payment has the same behaviour as cancelling an order. If you want everything to have the same behaviour, set the config option to not set items status to be in stock on order cancellation.
Credit Memo
When crediting a credit memo, if you choose the "Return to Stock" option, this will only increase the products stock quantity, it won't set the items stock availability (or stock_status) to "In Stock", if it is currently "Out Of Stock". This means that if someone purchases an item, putting it out of stock (perhaps if it only has 1 in stock), then crediting and returning to stock won't bring the item back in stock, it will simply increase the stock quantity.
Regarding the above, setting "Automatically Return Credit Memo Item to Stock" in the inventory config misleadingly unticks the "Return to Stock" checkbox on the credit note, making you think that all this config setting does is to set this checkbox to disabled by default. What this should do is to disable the box, because with this config setting enabled the stock return takes effect regardless of the checkbox state.
The code for the credit memo return to stockgets kicked off from an observer - Mage_CatalogInventory_Model_Observer::refundOrderInventory()
Order Cancellation
What is confusing is that there is a inventory config option called "Set Items' Status to be In Stock When Order is Cancelled", which actually does set the item to be "In Stock" when an order is cancelled. The annoying thing about this is that I can understand why Magento would refuse to ever automatically set something back to "In Stock", because you might have set it "Out Of Stock" on purpose, so Magento could never know if refunding or cancelling an order should set the item to be "In Stock", however having the ability to do this on cancellation, but not refund, makes no sense.
Code-wise, cancelling an order calls Mage_Sales_Model_Order::cancel(), which dispatches the sales_order_item_cancel event, which causes Mage_CatalogInventory_Model_Observer::cancelOrderItem() to run.
You can't stop order cancellation from returning the item to stock (i.e changing the stock quantity).
Payment Deny
This triggers an order cancellation, so has the same behaviour as above.
Note - Automatic Back In Stock Flag
There is a column named stock_status_changed_auto in `cataloginventory_stock_item' which indicates whether the last setting of "In Stock" was done automatically or not.
