Postgresql memory usage

I'm trying to optimize my database (its quite heavy), I have external server only for it with 6GB RAM. I was looking for settings and saw that WORK_MEM is setted to 36MB. I thought it is possible just to increase it to like 1GB and system will run much faster. But I was surprised to see that postgresql already use all of memory. How is it possible, there is another properties for using memory, like cache or something? Can I increase WORK_MEM despite fact that memory is already used?