People never seem to get this so I'll repeat it: The amount of memory only impacts performance when you run out. Up until you saturate 4 gigs there will be no difference between having 4 or 400 GB of system memory. You may see minor variations due to timings of individual units of memory or dual vs single channel configurations, but there's zero advantage to extra empty memory from a performance perspective.
Whether or not you need more RAM is determined entirely by what your normal usage consumes, so do some experimentation and figure it out. Remember that some of the system memory will be used as VRAM so if you're right up against 4GB it would be advisable to go with 8.
Honestly this sounds more like you went from a system with a spinning HDD to an SSD.